Hybrid Utrasound and MRI Acquisitions for High-Speed Imaging of Respiratory Organ Motion

Abstract

Magnetic Resonance (MR) imaging provides excellent image quality at a high cost and low frame rate. Ultrasound (US) provides poor image quality at a low cost and high frame rate. We propose an instance-based learning system to obtain the best of both worlds: high quality MR images at high frame rates from a low cost single-element US sensor. Concurrent US and MRI pairs are acquired during a relatively brief offine learning phase involving the US transducer and MR scanner. High frame rate, high quality MR imaging of respiratory organ motion is then predicted from US measurements, even after stopping MRI acquisition, using a probabilistic kernel regression framework. Experimental results show predicted MR images to be highly representative of actual MR images.

1 Introduction

Magnetic Resonance (MR) imaging has gained considerable traction in the last two decades as a modality of choice for image-guided therapies [1,2], primarily due to its excellent soft-tissue contrast and its non-invasive nature. However, major challenges include relatively slow frame rates and limited physical patient access within the MR bore. Perhaps the most notable effort made toward scanner design and providing patient access has been the (now discontinued) double-doughnut 0.5 T SIGNA SP/i design [3], whereby the interventionist could step in-between two physically-separate magnets and gain direct access to the patient. Other interventional MR systems have also been developed and commercialized but patient access, MR-compatibility of instruments and overall costs have remained considerable hurdles. In contrast to MR imaging, ultrasound (US) imaging provides fast frame rates and nearly-unhindered physical access to the patient. US imaging systems are cheaper and faster than MR, yet produce images that are often found lacking in terms of contrast and overall quality. As a consequence, several noteworthy efforts have been made to combine the two complementary imaging modalities and a body of work has emerged on developing hybrid methods [4,5,6,7,8,9,10]. In [4,5], hybrid 2D US/MR systems were proposed where orientation information extracted from US data was used to update the image slice position of an SSFP sequence in real time, for prospective motion compensation in a motion phantom. A similar hybrid system was presented in [7], where a clinical US imaging system was integrated with 1.5 T and 3 T clinical MR scanners for simultaneous 4D MRI and US imaging.

The present work involves a small 8 mm-diameter single-element MR-compatible ultrasonic transducer applied to the skin of the abdomen and held in place using a simple adhesive bandage (Figure 1). A regular flexible MR coil array can readily be wrapped over this small US probe, at no detectable penalty in MR image quality. The emitted US field is not focused, it is expected to penetrate and reflect possibly several times within the abdomen. A-mode ultrasound raw data signals (USrd) are acquired at a very high frame rate during regular MR image acquisition, and these signals act as a unique signature for the internal organ configuration, including respiratory state. This is in sharp contrast with more traditional US imaging whereby the imaging probe would consist of an array of transducer elements, hand-held over the anatomy of interest to capture images. While the simple and convenient USrd sensor used here is insufficient to produce spatially resolved US images, it provides a 1D trace rich in information that can be correlated with simultaneously-acquired and spatially-resolved MR images. It was previously shown that such a 1D USrd signal might be suitable as a biometric navigator [8]. Here, we use a hybrid US-MR system to achieve two goals within the realm of abdominal imaging under respiratory organ motion: First, the temporal resolution of the MR image sequence is artificially boosted by orders of magnitude using an algorithm that learns from a stream of simultaneous MR and US data, providing the interventionist with a real-time view of abdominal organ motion. Second, after a learning phase the algorithm can be applied to the US data alone, allowing high-rate image reconstructions even when the patient is outside the scanner, thus offering a new take on the problem of intra-procedural imaging. A distinguishing feature of the proposed approach comes from the low cost of the US system and the simplicity of the generated US signal.

Fig. 1.

Overview of the hardware setup. Left: MR-compatible USrd sensor (top), placed on the volunteer using adhesive bandage (bottom). Right: The system operates in two modes. First, the subject is placed inside the MR scanner for combined MRI and USrd acquisition (top). In this mode the system estimates MR images at high-speed while still learning from incoming MR data. In scanner-less mode, only USrd data are acquired but high-speed MR images are still synthesized based on the data from the training phase. No MR scanner is required in this case.

2 Materials and Methods

2.1 Hardware Setup and Data Acquisition

An MR-compatible, single-element USrd sensor (Imasonics, 8-mm diameter, 5.8 MHz) was inserted into a specially-carved rubber disc (3.5 cm diameter, 1.4 cm thickness), positioned onto the abdomen of the subject and held in place using an adhesive bandage (Walgreens, bordered gauze 10.2×10.2 cm). MR images were acquired using a 3 T General Electric system (Signa HDxt Twin Speed, 40 mT/m, 150 T/m/s) and a regular 8-element flexible cardiac array coil. Every TR interval, the MR scanner generated a trigger pulse for a pulser/receiver (Olympus 5072) to fire the USrd transducer. The resulting USrd data were recorded on a server via a sampling card (NI PCI-5122, National Instruments). The server also fetched the acquired MR raw data via a product raw-data server connection. Both incoming data streams, MR and US, were time-stamped and saved for processing. Figure 1 shows an overview of the experimental setup. Three human subjects were recruited and imaged following informed consent, over the course of four distinct scanning sessions (i.e. one of the subjects volunteered for two sessions). For the first three sessions a simpler MR protocol involving a single sagittal (S) plane was employed, while for the last session a more involved sagittal-coronal (S+C) two-plane protocol was employed instead (75 % partial-Fourier, two-fold parallel imaging scheme). Imaging parameters are listed in Table 1 and in the following: Flip angle = 30°, matrix size = 128×96 or 128×128 (S) / 192×192 (S+C), slice thickness = 5 mm, FOV = 20 cm (S) / 38 cm (S+C).

Table 1.

Overview of acquired datasets. S means ’sagittal’ and C means ’coronal’

Id	Mode	Acquisition length [s]	TR [ms]	fI [Hz]	fUS [Hz]	Number of MR images	Number of US traces
1.1	S	64	7	1.5	142.9	95	9120
2.1	S	198	7	1.5	142.9	295	28 320
3.1	S	116	10	0.8	100	91	11 520
4.1-4.2	S	109-122	8-10	0.8-1.3	100-125	95-155	12160-14880
5.1-5.6	S+C	50-111	18	1.7	55.6	86-192	2624-6144

2.2 Algorithm - Simpler Case: Single-Plane MR Acquisition

Typical US image reconstruction algorithms, based on a delay-and-sum beam-forming operation, are designed to discard much of the received US signals as they may have failed the basic US spatial encoding process; in contrast, the proposed algorithm utilizes raw US data without discarding any of them. While there is nothing special about the MR or the USrd signals obtained here, interesting behaviors appear as correlations are found between them, allowing USrd signals to become a surrogate for MR data. Given enough data is acquired for these correlations to be learned from, it becomes possible to boost the temporal resolution of a sequence of MR images by orders of magnitude. The overall proposed approach is made particularly interesting by the fact that the generation of MR-like images can be continued even after the subject is taken out of the MR scanner, solely based on the USrd signal.

The majority of temporal models are based on the Markovian dependency assumption, i.e. the current state is dependent only on its immediate predecessor. Temporal modeling, either short term Markovian or cyclical motion, faces the difficulty of estimating motion in the presence of irregularities, such as irregular breathing, gasping or coughing. Instance-based learning (IBL), on the other hand, operates by storing a potentially large number of training data samples in memory, then performing inference on new data based directly on previously-seen instances. This non-parametric approach is fundamentally different to training a parametric model from data instances and has many advantages: It can be adopted in situations where parametric models are unknown or difficult to specify accurately and it scales to the granularity of the data space. Furthermore, it is known [11] that IBL is in probabilistic terms equivalent to averaging over all the (possibly infinitely many) models of a fixed model family. Thus, IBL will become increasingly relevant as data acquisition, storage and retrieval systems increase in size and speed.

Let U_t represent the observed ultrasound vector at time t and let I_t be a random variable representing the estimated MR image at time t. Let further D = {I, U} be the collection of all previously acquired MR and US data I and U, respectively. Our method seeks to estimate I_t from U_t, i.e. to estimate an MR image for each USrd signal coming in at a frequency much beyond that of the MR image acquisition process. For this purpose we propose computing the expected value of I_t conditioned on U_t,

$$\underset{I_t}{\overset{𝔼}{}}[I_t\mid U_t,D]=\int I_{t}p(I_t\mid U_t,D)d{I}_t=\frac{\int I_{t}p(I_t,U_t\mid D)d{I}_t}{p(U_t\mid D)}.$$ (1)

The second equality results from applying the Bayes rule, where p(I_t, U_t|D) is the joint density of observed US trace U_t and MR image I_t, conditioned on previously seen data D. We propose an instance-based method for computing Equation (1), as follows. The joint density in the numerator is estimated using Kernel Density Estimation (KDE)[12] of the form

$$p(a,b)\approx \frac{1}{N}\sum _i{k}_a(a-a_i)k_b(b-b_i).$$ (2)

Let {(I_i, U_i)} be a set of training instances, consisting of N concurrently acquired MRI and US pairs (I_i, U_i). As a modeling choice, we define k_a to be the Dirac delta function centered at I_i and k_b to be a Gaussian centered around U_i with isotropic covariance matrix Σ = c · $𝕀$, where $𝕀$ is the identity matrix. The numerator in Equation (1) then becomes

$$\int I_{t}p(I_t,U_t\mid D)d{I}_t\approx \frac{1}{N}\int I_t\sum _i\delta (I_t-I_i)\mathcal{N}(U_t;U_i,\Sigma )d{I}_t$$ (3)

$$=\frac{1}{N}\sum _i{I}_i\mathcal{N}(U_t;U_i,\Sigma ).$$ (4)

The normalizing factor p(U_t|D) in the denominator of Equation (1) is unimportant in maximum a-posteriori estimation, however it is required for computing the expectation. It can be estimated in a similar manner to the numerator as follows:

$$p(U_t\mid D)\approx \frac{1}{N}\sum _i\mathcal{N}(U_t;U_i,\Sigma ).$$ (5)

Combining Equations (4) and (5), the final computational form of the estimator in Equation (1) becomes

$$𝔼[I_t\mid U_t,D]\approx \frac{{\Sigma }_i{I}_i\mathcal{N}(U_t;U_i,\Sigma )}{{\Sigma }_i\mathcal{N}(U_t;U_i,\Sigma )}.$$ (6)

Note that the forms of the numerator and denominator in Equation (6) are equivalent to Nadaraya-Watson Kernel Regression [13].

Evaluating $𝔼[I_t\mid U_t,D]$[I_t|U_t, D] requires a computationally expensive sum over a potentially large number of training samples. However with a suitably small c, the sum is dominated by small number of ’nearest neighbors’ for U_t. A search can performed to identify a set of k-nearest neighbors {U_i} of the current US observation U_t within the training data set. This search can be performed efficiently with fast approximate search methods, e.g. k-d-trees. Equation (6) can then be computed from elements within this subset. Note that with k = 1, the expectation is equivalent maximum a-posteriori estimation, which in our experiments leads to noisy estimates in the case of discrete data. Computing the expectation with k > 1 leads to smoother estimates of I_t via considering a weighted average of similar observations. Note also that a low posterior probability generally indicates an outlier U_t that has not previously been observed, which can be used to detect unexpected organ configurations.

2.3 Algorithm - Extension to Multiple-Plane MR Acquisition

An interesting extension to the algorithm is to work on MR sequences that acquire multiple intersecting slices and to estimate a coherent volumetric image. This means the probabilistic formulation must be adapted to take into account how well the matches of different planes agree. For the simplest case of two alternating slice positions, the expectation now becomes

$$\underset{I_t,J_t}{\overset{𝔼}{}}\left[\right[I_t,J_t]\mid U_t,D]=\int \int [I_t,J_t]p(I_t,J_t\mid U_t,D)d{I}_{t}d{J}_t,$$ (7)

where [I_t, J_t] is a concatenation of the two images.

Let [I_t]_L and [J_t]_L be all pixels from I_t and J_t, respectively, that are located at the intersection between the two planes. ${\left[I_t\right]}_{\stackrel{‒}{L}}$ and ${\left[J_t\right]}_{\stackrel{‒}{L}}$ represent all other locations. The joint distribution p(I_t, J_t, U_t|D) can be factorized into conditionally independent regions of I and J given U_t, and a dependent region where they intersect,

$$p(I_t,J_t\mid U_t,D)=p({\left[I_t\right]}_{\stackrel{‒}{L}}\mid U_t,D)p({\left[J_t\right]}_{\stackrel{‒}{L}}\mid U_t,D)p({\left[I_t\right]}_L,{\left[J_t\right]}_L\mid U_t,D).$$ (8)

We model p([I_t]_L, [J_t]_L|U_t, D) with a Gaussian $\mathcal{N}({\left[I_t\right]}_L;{\left[J_t\right]}_L,{\Sigma }_L)$ with empirically-determined covariance matrix Σ_L and arbitrarily choosing [I_t]_L (or [J_t]_L) as the mean. Putting everything together and again applying KDE as done in Equation (6) leads to

$$\underset{I_t,J_t}{\overset{𝔼}{}}\left[\right[I_t,J_t]\mid U_t,D]\approx \frac{{\Sigma }_i{\Sigma }_j[I_i,J_j]\mathcal{N}(U_t;U_i,\Sigma )\mathcal{N}(U_t;U_j,\Sigma )\mathcal{N}({\left[I_t\right]}_L;{\left[J_t\right]}_L,{\Sigma }_L)}{{\Sigma }_i{\Sigma }_j\mathcal{N}(U_t;U_i,\Sigma )\mathcal{N}(U_t;U_j,\Sigma )\mathcal{N}({\left[I_t\right]}_L;{\left[J_t\right]}_L,{\Sigma }_L)}.$$ (9)

where U_i and U_j are the USrd signals corresponding to images I_i and J_i, respectively. The generalization to an arbitrary number of image planes parallel to I_t and J_t is straight forward by repeated application of the product rule in Equation (8). Allowing for arbitrary plane orientations involves higher-order terms but follows the same principle.

3 Results and Discussion

Figure 2 shows examples of estimated images for times when an MR image was also acquired. This allows to compare the estimates to their ground-truth images. As the figure shows, estimates at both exhale and inhale are very similar to the actual MR images acquired. In order to better visualize time, Figure 3 shows plots of a single line of pixels in superior-inferior direction over time (m-mode) for dataset 6.1. At the beginning, neither the acquired images nor their predictions reflect the temporally highly-resolved USrd signal. However, after a short training period the estimated MR sequence runs at the same speed as the ultrasound. Notably, after the MR acquisition is stopped, the algorithm continues to give estimated MR frames that are very well in agreement with the USrd signal. For datasets 1.1-4.2, the location of a clearly visible vessel was manually selected in 10 MR images after one minute of learning. An average error of 1.19 px (standard deviation 0.8 px) was determined, which shows that the algorithm in fact estimates MR images that accurately represent respiratory motion.

Fig. 2.

Comparison of acquired MR images and their estimates.

Fig. 3.

M-mode visualizations of dataset 6.1. The image on the right shows the position of the m-mode line.

The proposed system could potentially be used to track lesions during image-guided therapy. Compared to MR, it is extremely simple and cheap; even so, it allowed temporal resolution to be improved by orders of magnitude compared to MR alone. Limitations of the present study included the small number of human subjects recruited so far, the fact they were healthy volunteers rather than patients, and the off-line nature of the currently-implemented reconstruction chain. While all aspects of the processing did execute fast enough on an off-the-shelf PC to be compatible with a real-time application, individual components have not yet all been implemented and linked for truly real-time reconstruction and display to occur. Future work further includes the development of problem detectors to identify and gracefully handle all time periods when motion might momentarily become too rapid and unpredictable, such as during violent coughing or gasping.