Super-resolution ultrasound imaging

Abstract

The majority of exchanges of oxygen and nutrients are performed around vessels smaller than 100 um, allowing cells to thrive everywhere in the body. Pathologies such as cancer, diabetes or arteriosclerosis can profoundly alter the microvasculature. Unfortunately, medical imaging modalities only provide indirect observation at this scale. Inspired by optical microscopy, ultrasound localization microscopy has bypassed the classical compromise between penetration and resolution in ultrasonic imaging. By localizing individual injected microbubbles and tracking their displacement with a subwavelength resolution, vascular and velocity maps can be produced at the scale of the micrometer. Ultrasound super-resolution has also been performed through signal fluctuations with the same type of contrast agents, or through switching on and off nano-sized phase change contrast agents. These techniques are now being applied pre-clinically and clinically for the imaging of the microvasculature of the brain, kidney, skin, tumors and lymph nodes.

Introduction

Our circulatory system is so vital that the loss of blood flow is one of the key parameters defining death. This vessel network created by nature, comprises billions of vessels that carry fundamental nutrients, hormones and gases at distances larger than simple diffusion in large living beings (Pugsleya et al, 2000). Would one lay out all of the arteries, veins, and the 40 billion capillaries in one human adult, they would reach more than 100 000 km or two times the circumference of the earth. The tiniest components of our vasculature, the capillaries, are less than 10 μm in diameter (Lenasi et al. 2016) or about a tenth of the diameter of a human hair. Some capillaries are even smaller in diameter than blood cells, forcing cells to distort their shapes to pass through.

From a biomechanical point of view, the circulatory system is also a piece of extraordinary machinery ensuring a rapid transport and complete distribution of blood at meters per second in our largest arteries down to less than one millimeter per second in the capillaries feeding the vast territory of our organs at microscopic scales. To achieve this amazing feat, the three-dimensional geometry of our vasculature and the rigidity of each arterial segment are carefully optimized. The arterial stiffness also adapts itself transiently following a load or arterial pressure changes. This highly nonlinear elastic nature of the arterial walls is essential to effectively damp the large oscillations in blood flow coming from the heart. It provides for a better flow homogeneity in tiny blood vessels distal in the arterial vascular tree. As a consequence, pathological modifications of the mechanical properties of arteries strongly affect the correct transmission of blood to tiny vessels (Webb et al 2012). Today, although the field of regenerative medicine and biomaterials is fast progressing, mimicking the complete vascular system with optimal structural and functional properties remains challenging.

In some noble organs such as the brain, the complexity goes even at a higher level, as tiny vessels are intimately connected and communicating with the neuronal system via the glial system and particularly astrocytes and pericytes (Ladecola et al. 2004). Such communication allows for precise coupling between neuronal activity and blood flow, which insures that activated brain regions are properly nourished in oxygen, glucose and other nutrients. This neuro-vascular coupling is the basis for functional magnetic resonance imaging (Kim and Ogawa, 2012) and functional ultrasound (Deffieux et al. 2018). In the brain, as in other organs, the microvasculature is a dynamic system that adapts to the constantly changing metabolism of surrounding cells.

At such a microscopic level, large territories of our knowledge remain unexplored mainly due to the lack of imaging methods providing non-invasiveness, microscopic resolutions, a macroscopic field of view and sufficient temporal resolution for dynamic imaging. From a fundamental point of view, it is, for example, striking to note that scientists do not fully understand the functioning of the human placental vascular system and exchanges between maternal and fetal blood systems (Mayo et al 2018).

Many discoveries have also shed new light on the major importance of our vascular system in various disease processes, ranging from cancer, diabetes to neurodegenerative diseases such as Alzheimer or Parkinson diseases (Stanimirovic et al 2012, Zlokovic et al 2011). For example, it has been known for more than 40 years that angiogenesis, the development of new blood vessels, is a hallmark of solid tumors (Folkman 2019). Angiogenesis is driven by tumors that outgrow their host tissue’s native blood supply, with the result of the release of pro-angiogenic factors that locally stimulate increased microvascular development to feed the growing malignancy. Pathological angiogenesis is differentiated from the normal microvascular structure by the lack of hierarchical branching, the presence of tortuous and erratically shaped vessels, and immature and leaky vessels (Jain 1988).

Dementia also has a microvascular component, as it is more likely to be present when vascular and Alzheimer disease lesions coexist (Jellinger et al 2008). In elderly individuals, the association between stroke and AD increases in patients with vascular risk factors (Iadecola et al 2010, Gorelick 2011). Vascular endothelial growth factor (VEGF), one of the most potent mediators of angiogenesis, can be envisioned as potential therapeutics for neurodegenerative disorders (Storkebaum et al 2004).

The diagnosis of these diseases would benefit greatly if the microcirculation could be characterized in each tissue, organ, and patients. Even if one could question the relevance of this longstanding and infinite quest for finer spatial resolution in medical imaging, assessing the architecture and functioning of always smaller vessels means going backward in time on the disease. Indeed, pathologies leave first their signature in tiny vessels before becoming detectable much later in larger vessels by a dramatic domino game.

Unfortunately, the naked eye cannot resolve the vessels smaller than 100 microns forming the microcirculation. Moreover, most of these vessels lie beyond the penetration depth of coherent light in tissue. Histopathology can be performed following a biopsy or a surgical resection and remains a gold-standard for cancer diagnostic for instance. However, such approaches are limited by their invasiveness in the clinical setting or preclinical research. Microvascular parameters linked to angiogenesis such as the microvascular density or the intercapillary distance are obtained by observing and measuring stained vessels on highly-magnified thin slices under a microscope. Several types of tissue staining can help reveal microvessels specifically such as anti-CD31, CD34 or von Willebrand factor (Weidner 1995, Marien et al. 2016).

Within a few hundred micron depth, microscopy can also be applied directly on the skin or mucosa to observe its microcirculation. Techniques such as orthogonal polarization spectral imaging or sidestream dark-field imaging can extract specifically the light from blood and provide a map of blood vessels under the surface (Leahy 2012). Flow can also be assessed with laser Doppler either at a single point or on an entire map. Additionally, retinography can assess the evolution of the microcirculation, in diabetic patients for example, by exploiting the clear window provided by the eye (Pieczynski et al. 2015).

Various modalities are able to reach microscopic resolutions such as 2-photon imaging (Soeller et al 1999) or optical coherence tomography (Jia et al 2012) at the cost of a limited field of view and penetration depth. Other approaches based on tissue clearing coupled to light-sheet microscopy lead to high-resolution volumetric imaging of the microvasculature in organs but are limited to dead tissues (Ragan et al 2012, Ertuk et al 2012). Additional approaches such as photo-acoustics (Wang et al 2012) or functional ultrasound (Deffieux et al 2018) based on ultrafast Doppler (Bercoff et al 2008, Demene et al 2016) recently improved our ability to image small vessels (of the order of 100 μm diameter) but fail to reach microscopic resolution scales.

Moreover, for human diagnostic or animal imaging in-depth, it is necessary to exploit modalities that can explore entire organs, at a depth beyond 10 cm. Well-known medical techniques, such as MRI (Williams et al. 1992), CT (Miles 1991), nuclear imaging (Underwood et al. 2004) and ultrasound (Cosgrove and Lassau 2010) all have versions that are sensitive to blood perfusion. Perfusion CT works by following a bolus of injected iodinated material in different parts of the organ. Parameters such as blood flow, blood volume, time to peak and mean transit time can be extracted from the bolus curves. In a similar fashion, perfusion magnetic resonance imaging can also be performed with a contrast agent such as gadolinium chelate complex. Furthermore, using arterial spin labeling, MRI can even provide information on perfusion without contrast agent injection (Peterson et al. 2006). With the use of injected radionuclide, single-photon emission computed tomography can be made sensitive to perfusion, that of the cardiac muscle for instance. However, these techniques provide only broad generalities on the microcirculation in each imaging voxel. None of them can define the microvascular architecture itself because these macroscopic modalities are limited in resolution to the submillimetric and millimetric scale.

In particular, ultrasound imaging is limited in resolution by diffraction to the scale of its wavelength (wavelength=speed of sound/frequency, 5 MHz ultrasound wave in tissue has a 300-micron wavelength). It relies on the echo of tissue due to variation in compressibility and density to reconstruct anatomical images. In Doppler mode, it is sensitive to blood vessels through the motion of red blood cells serving as scatterers. However, small vessels have a limited number of weak scatterers, which are also moving slowly, making it particularly difficult to distinguish vessels from tissue motion. Generally, vessels with blood velocities below 1 cm/s are difficult to distinguish, making Doppler ultrasound a poor imaging method for the microvasculature. Even with recent advances exploiting ultrafast plane-wave imaging and spatiotemporal filters (Bercoff et al 2008, Demene et al 2016), which improved drastically the sensitivity of Doppler ultrasound, microarterioles and microvenules remain invisible to Doppler ultrasound.

As in other medical imaging techniques, ultrasound imaging can be made sensitive to unresolved microvessels by the introduction of contrast agents (Cosgrove and Lassau 2010). These agents are microbubbles, smaller than capillaries, which are injected intravenously and flow within the entire vasculature for about 3 minutes (Blomley et al. 2001, Ferrara et al. 2000, Burns et al. 2006). Microbubbles act as resonators with a resonance frequency in the 1–15 MHz range, vastly increasing their scattering coefficient in the clinical frequency range. Moreover, microbubbles re-emit ultrasound in a nonlinear fashion, providing a tool to separate them from tissue (Frinking et al. 2000, Dollet et al. 2008, Stride et al. 2003). The presence of these contrast agents highlights the vasculature, including the capillaries, as the ultrasound scanner is also sensitive to slowly moving microbubbles. One great advantage of microbubbles for perfusion imaging is that they are entirely intravascular due to their micrometer size. Hence, after injection, the only compartment to be taken into account for the calculation of parameters such as the mean transit time is the vasculature. Crucially, in contrast to optical agents, microbubbles can be detected deep within the body, making them advantageous as a clinical contrast modality. Furthermore, contrast-enhanced ultrasound (CEUS) scans are already an established and routine clinical procedure in many clinics around the world, making the fast clinical translation a real possibility.

Unfortunately, conventional ultrasound remains limited by resolution in the same way as MRI, CT or SPECT. The extracted parameters are linked only indirectly to modifications in the microcirculation. If a medical imaging technique could provide a direct mapping of microvessels, it would provide a revolutionary wealth of information, bridging the gap with histopathology. For instance, such a technique could measure directly the vessel density, their inter distance, size, unique flow pattern, tortuosity or fractal factor.

The extensive work on microbubble imaging has recently inspired a new technique that has caused an important rupture in a fundamental characteristic of ultrasound: its resolution. Introduced 10 years ago, ultrasound super-resolution can improve the resolving power of ultrasound imaging by a factor of 10 with respect to the diffraction limit (wavelength/2).

This review describes ultrasound super-resolution as it is conceived by several groups which introduced several of the precursor work on the field. It will first detail its origin and its technical aspects along with defining its key concepts and technical aspects. It will detail both ultrasound localization microscopy and other approaches based on fluctuations imaging. The latter section will detail the current and future applications for oncology and neurology.

The origin of ultrasound super-resolution

Ultrasound super-resolution imaging has been discussed for several decades (Ikeda et al 1979, Jones et al. 1992, Couture et al. 2018). The goal of super-resolution is to separate echoes coming from sources closer than the classical diffraction limit. Such a quest was performed in parallel to the improvement of resolution through the increase in acquisition frequency (Lockwood et al. 1992).

Approaches such as near-field imaging (Shekhawat et Dravid 2005) were shown to differentiate subwavelength sources since the resolution close to a probe is proportional to the distance with respect to the object rather than the wavelength (Fink and Tanter 2010). However, in the body, organs are several centimeters deep, which could be a hundred wavelengths away. A far-field approach is thus required for medical imaging.

In the far-field, refocusing on close individual sources could be performed when a limited number of them were present (Prada et Thomas, 2003, Blomgren et al 2002, Lehman et al 2003). A precise knowledge of the source could also allow subwavelength imaging (Clement et al. 2005). But a limited number of sources or strong a priori knowledge are not applicable to conventional B-mode imaging, which observes tissue formed by a multitude of scatterers at various scale: cells, organelles, fibers, etc.

Further rupture of the half-wavelength limit in ultrasonic imaging was inspired by new developments in optical microscopy. In 2006, F-PALM, PALM, and STORM were introduced, breaking the diffraction limit in optics by at an order of magnitude or more (Betzig et al. 2006, Hess et al. 2006, Rust et al 2006). It relies on photoswitchable fluorescent sources and fast cameras which take sequential images where only a subset of the sources is lit in each image. By isolating the sources closer to the wavelength, the interference of the wave they emitted could be avoided. Moreover, the knowledge of the point-spread function of the system lead to an extremely precise localization of isolated source from their intensity map. By collecting thousands of subwavelength localizations, a picture with a resolution in the tens of nanometer could be obtained with a microscope using visible light. These developments were so revolutionary that they led to the attribution of the 2014 Chemistry Nobel Prize to Eric Betzig, Stefan Hell and William E. Moerner.

In 2010, an ultrasonic version of FPALM, now called Ultrasound Localization Microscopy (ULM), was proposed (Couture et al. 2010). The fluorescent beacons were replaced by ultrasound contrast agents and the cameras by an ultrafast programmable ultrasonic scanner (Couture et al. 2009, 2012). Beyond that, the same principle applied, the interference between different microbubbles was avoided by observing them sequentially so that isolated sources could be detected in each image. The point-spread function on the radio-frequency channel data or on the beamformed image being known, a localization with a micrometric precision could be obtained for each microbubble. Since these contrast agents are purely intravascular, the accumulation of these subwavelength localizations would yield a super-resolved map of the microvasculature.

The ULM approach was rapidly demonstrated in vitro through imaging a single micro-channel containing flow microbubbles (Couture et al. 2011). In parallel, the first in-vivo application was reported by Siepmann et al. (2011), who described a technique to improve maximum intensity projection images of dilute microbubbles by implementing centroid detection. By 2013, four of our teams were already exploring super-resolution imaging. In-vitro, Viessman et al. (2013) showed for the first time that super-resolution can be achieved, i.e. two vessels separated by less than half a wavelength could be distinguished using ultrasound localization microscopy. Two 3D super-resolution approaches were proposed, one with a 1.5D array (Desailly et al. 2013) and another with a hemispherical array through human skull bone (O’Reilly et al. 2013).

Since microbubbles and ultrasonic scanner were already clinically in use, in-vivo applications were rapidly implemented afterward. In particular, Christensen-Jeffries et al. (2015) demonstrated the application in the mouse ear, Errico et al. in the rat brain (2015), Lin et al. in a cancer model (2017), and the first in human demonstration of the techniques by Opacic T (2018) and Harput S (2018).

These applications will be detailed in the latter sections of this review. At this point, it is important to explain precisely the principle of ultrasound localization microscopy. Non-ULM approaches will also be introduced.

General technical aspects of ultrasound localization microscopy

Like optics, ultrasound faces a limit inherent to all wave-based imaging processes, where diffraction of the transmitted and received waves mean point sources become indistinguishable from one another when closer than approximately half the transmitted wavelength. Beyond this, interference of scattered sound results in acoustic speckle. Following the revolutionary developments seen within the optical field, analogous approaches were proposed to exploit these same principles, but in the ultrasound field. Here, instead of utilizing molecules to provide the individual signal sources required, ultrasound contrast agents, called microbubbles, were proposed.

Thus, the ultrasound super-resolution process requires the introduction of a contrast agent into the body. Akin to its optical counterpart, it also needs the acquisition of a sequence of frames. A crucial principle within localization microscopy techniques is that by limiting the number of sources detected in each image, the responses do not interfere with each other. Under this constraint, the location of the underlying scatterers, in this case, microbubbles, can be estimated to a precision far higher than the diffraction-limited resolution of the system. Here, this is exploited to accumulate the localization of flowing microbubbles over thousands of images to recreate a super-resolved image of vascular structures.

Increased worldwide attention within this area of research means the contribution of many international groups is facilitating rapid progression, diversity, and innovation in this field, which will inevitably encourage and accelerate clinical implementation. Despite methodological differences, there are a number of common steps that form the basis of the technique throughout the literature. These are (note: post-processing steps are visualized in Figure 1):

Figure 1.

Steps in ultrasound super-resolution processing. A) Acquisition of ultrasound data over time from contrast-enhanced vascular region. B) Detection of signals from microbubble contrast agents. C) Isolation of individual microbubble signals, overlapping or interfering signals are rejected. D) Localization of the microbubble to a precision far beyond the diffraction-limited resolution. E) Tracking of the microbubbles through consecutive frames to establish velocity profiles. F) Mapping of the accumulated localizations gathered over the series of frames produces an image of the vascular structure far beyond the diffraction limit.

1). Microbubble Introduction

All current single-bubble localization methods for super-resolution involve the injection of contrast agents as a bolus or infusion. For localization-based methods, the concentration of the agent is required to be low enough that bubbles can be spatially separated by the image system’s diffraction-limited point spread function following post-treatment filtering. Recent sparsity-based approaches (Bar-Zion et al. 2018) and deep learning based methods (van Sloun et al. 2018) can alleviate these requirements and permit higher concentrations

2). Video Acquisition

An ultrasound pulse is emitted into a medium containing microbubbles. A video acquisition of microbubble flow is then acquired (Fig 1A). This could be B-Mode with or without contrast-specific pulse sequences, and at conventional or ultrafast frame rates. The received data can be collected as a matrix of radiofrequency (RF) data acquired by each channel, or as beamformed image data. Differences may depend on equipment availability, restrictions with data accessibility, or specific clinical protocols.

3). Motion correction.

Long video acquisitions are often required to observe the smallest vessels. Since super-resolved images are created from the superposition of many localizations gathered over time, motion between frames will significantly affect their visualization. Crucially, the scale of motion present in clinical scanning is often orders of magnitude larger than the super-resolved vessels themselves. Adequate motion correction techniques are therefore vital.

4). Microbubble detection

A crucial part of the super-resolution process is distinguishing microbubbles from the surrounding tissue (Fig 1B). This step creates candidate bubble regions for subsequent localization. Inadequate bubble detection will mean long acquisition times are needed to obtain sufficient localizations for the final rendering. Too many erroneous signals may create a challenge for subsequent filtering processes and result in noise in the super-resolved image. Techniques to extract microbubble signals from tissue have been thoroughly studied in the field of contrast-enhanced ultrasound (Cosgrove and Lassau 2010), while new techniques are also being introduced and developed within the ultrasound super-resolution field itself.

5). Microbubble Isolation

Once detected, a filtering step can identify isolated microbubbles in each frame (Fig 1C). Indeed, the echoes from two microbubbles that are closer than a few wavelengths in an image will interfere, making the corresponding localizations of each microbubble inaccurate. These signals should, therefore, be rejected at this stage or at the tracking stage. Consequently, only a limited number of microbubbles can be detected in each image to avoid such overlapping.

6). Localization

A key step in the process is the localization of these isolated signals (Fig 1D). Ultrasound waves are coherent when propagating into tissue. Furthermore, the response of ultrasound waves to a single isolated point scatterer is generally well defined by the point spread function of the imaging system. Under the assumption, these signals are isolated, this procedure can estimate the location of the underlying scatterers to a precision far higher than the diffraction-limited resolution of the system. Localization of the microbubbles can be performed with a number of techniques in the RF domain or on beamformed images. In reality, the ultimate limit on the resolution in a super-resolved image is determined by the precision of this localization step.

7). Tracking

By determining the displacement of a microbubble between two images, a velocity vector can be created (Fig 1E). With sufficient detections, features of the local vascular velocity can be determined through the creation of detailed velocity maps. The tracking of these localized points to define the paths and velocity of microbubbles in microvessels can vastly improve the quality and information content of images and modify its interpretation. Such microbubble tracking has distinct advantages over Doppler in spatial resolution and is largely independent of the direction of the flow displacements.

8). Visualization

The visualization of localizations, their density, or their calculated velocities is performed by creating a map from the microbubble positions after correcting for motion to reveal details of the vascular network (Fig 1F). This can be performed either by accumulating bubble detections in fine grids with pixel sizes that reflect the scale of the super-resolved detail or plotting each localization as a Gaussian distribution, where its size accounts for the uncertainty in the localization procedure.

Definitions

Ultrasound super-resolution is a new technique with many promises, but with terminology that could become confusing. One of the goals of this review is to establish a common language and provide several definitions. In particular to unite applications that can be widely different, such as between the simple improvement of vascular imaging in deep-seated tissue in humans or the direct observation of capillaries an animal brain.

Localization precision:

Ultrasound localization microscopy relies on the accumulation of the centroid’s localization of isolated microbubbles to reconstruct a super-resolved image of the microvasculature in which they flow. One of the key aspects is hence the capacity to localize a single source with a good precision either on the radio-frequency channel data or the beamformed image. The localization precision is the closeness between several measures of the localized position of a single isolated microbubble. It depends for instance on the SNR (Desailly et al. 2015). This precision is often close to the size of the microbubbles and that of the capillaries. This precision is not the resolution of the imaging system as the microbubbles are not the structure to be imaged, but they are rather the probes that highlight that structure.

Isolated microbubbles:

A source is deemed isolated if, in a single image, no other sources can bias significantly the localization of its center. This depends on the techniques that have been developed to distinguish non-isolated sources, along with the contrast-to-tissue ratio, the SNR and the point-spread function of the system.

Resolution:

In super-resolution ultrasound imaging, as in any imaging modality, resolution is the capacity at separating two objects to be imaged. But, in our case, these objects are the vessels to be imaged. The separation of two vessels or the capacity at separating two distinct flows patterns within the same vessel is the ultimate measurement of resolution in our field. Other approaches from the optical field should also be exploited such as the Fourier Shell Correlation threshold (Van Heel et al 2005).

Super-resolution:

Super-resolution is the capacity at distinguishing two objects, here vessels, beyond the classical limit. Several classical limits have been proposed and their relevance depends ultimately on the application. The lower-limit for resolution is usually considered to be the diffraction barrier at half the wavelength. One can also use the Rayleigh resolution criterion of 1.22λ(Focal length/Aperture) (Cobbold 2006). The latter definition is more lenient in the vast majority of cases, but providing a 150-micron resolution at 15 cm depth with a 5 cm aperture 5-MHz transducer, remains an interesting exploit for medical applications, especially for larger organs. In both cases, the particular limit should be clearly stated in the article to allow a better comparison between studies.

It should be noted post-filtering techniques on diffraction-limited images are not considered super-resolution as they assume a priori knowledge of the vessels to be separated. Indeed, prior information on the ultrasound echo sources has to be exploited before their accumulation, as it is their physical interference in each frame that has to be avoided to achieve super-resolution.

Ultrasound super-resolution imaging

Ultrasound super-resolution imaging refers to the field of study comprising several techniques — using high-frequency acoustic waves — that can distinguish objects or structures closer than the diffraction-limit, which is about a half-wavelength. These techniques include ultrasound localization microscopy, super-resolution fluctuation imaging, structured illumination or near-field approaches. Early work on the subject can be found in the 1970’s (Ikeda et al. 1979).

Ultrasound Localization Microscopy (ULM)

Ultrasound localization microscopy refers to an ultrasound super-resolution technique. It exploits the accumulation of subwavelength localizations of many separate sources over a great number of frames to reconstruct a super-resolved composite image. In each frame, the localization sources are sufficiently sparse so that their interference can be disregarded or taken into account to yield a precise positioning.

Temporal resolution:

Temporal resolution corresponds to the total time to display a super-resolved image. It has to be distinguished from the frame rate of the acquisition. The temporal resolution and spatial resolution are closely linked since the accumulation of microbubble localization will yield improved image. After injection, microbubbles remain rare compare to the red blood cells (about 1/100,000) and it takes time for these agents to fill every single capillary.

Microbubble differentiation vs tissue

The first stage for ultrasound localization microscopy is to separate microbubbles from tissue. Fortunately, since the first ultrasonic detection of microbubbles in-vivo (Gramiak and Shah 1968), several techniques have been proposed to highlight the signal of contrast agents. With respect to tissue or blood, microbubbles have several distinct characteristics.

First, these intravascular microbubbles move with the blood, allowing them to be separated from static or slowly moving tissue. The first microbubble detection technique was hence ultrasound Doppler. For instance, on continuous Doppler, isolated microbubbles in the blood make a very distinctive broadband “ploc” as they pass through the acoustic beam.

As discussed before, microbubbles are clearly distinct from the blood by the intensity of their echo. Their scattering cross-section can be three orders of magnitude greater than their physical size (Wheatley et al. 1990). This is partly due to the important acoustic mismatch between air and an aqueous solution. But it is also due to a very beneficial coincidence of nature. Indeed, microbubbles act as a resonator where the compressible air is the spring and the surrounding water is the mass. The resonant frequency of microbubbles small enough to move in the capillary falls exactly in the MHz-range within which clinical ultrasound imaging is performed. This resonance increases the echo of microbubbles by several orders of magnitude.

But beyond moving faster than tissue and scattering more than red blood cells, microbubbles can also be detected through their nonlinear echoes (Shi et al. 2000). Because of their strong oscillations around their resonance, the acoustic behavior of microbubbles goes beyond simply following the incident wave. In the scattered spectrum, this nonlinearity can be detected as a rich harmonic, subharmonic, superharmonic, and ultraharmonic content and microbubbles can be detected as such (Forsberg et al. 2000, Burns et al. 1994, Basude et al. 2001). However, multipulse sequences are even more sensitive to contrast agents and commercial systems have implemented techniques such as pulse-inversion (Simpson et al. 1999, Eckersley et al. 2005). Pulse-inversion, Amplitude modulation or a combination of both to highlight perfusion after microbubbles injection.

Finally, beyond a few hundred kPa, microbubbles oscillate so violently that they can disrupt, their shell rupturing and their gas diffusing in the surrounding fluid (Postema et al. 2004). The disappearance of the microbubbles after a short but intense pulse can be used as a contrast mechanism through differential imaging, but can also be applied for flash-replenishment techniques (Johnson et al. 1995, Greis 2009).

All these approaches were already exploited to detect microbubbles for ultrasound localization microscopy. Specific techniques depend generally on the frequency used, which itself depends on the depth at which the imaging is performed. At higher frequencies, microbubbles are poorly resonant and scatter little harmonics, so techniques based on the motion or disruption of microbubbles are preferable. For instance, spatiotemporal filtering with singular value decomposition has, importantly, increased Doppler sensitivity and has been exploited to separate microbubbles from tissue (Errico et al. 2015, Desailly et al. 2017, Song et al. 2017). In addition, deep learning techniques can be used to solve the microbubble tissue separation problem through an unfolded robust principal component scheme (Cohen et al., “Deep Convolutional Robust PCA with Application to Ultrasound Imaging”, ICASSP, 2019). More simply, differential imaging can also highlight the motion or disruption of microbubbles (Desailly et al. 2013).

Closer to resonance, it is highly beneficial to use nonlinear techniques. Pulse-inversion and amplitude modulation were all used to extract the signal from microbubbles (Viessman et al 2013) (Christensen-Jeffries et al, 2015, 2017). These various approaches were compared in a recent article by Brown et al. (Brown et al 2019).

For cardiac, abdominal, or transcranial applications, depths on the order of 10 cm are required. However, detecting separable microbubbles in microvessels at these depths is challenging because the intensity of the signal backscattered from microbubbles is low and increasing the transmit energy is not a viable option due to microbubble destruction. Previous studies have proposed the use of larger microbubbles to increase the amplitude of the backscattered signals (Lin et al 2017), however further improvements are needed to achieve super-resolution at clinically relevant depths. Furthermore, the accurate evaluation of microvascular structure requires imaging in 3-D where the small aperture sizes of existing matrix arrays are challenged in terms of sensitivity and contrast. However, the volume fraction of tissue that is occupied by vessels is typically less than 10%, therefore imaging sequences do not need to interrogate the entire volume.

A simultaneous multi-focus beamforming approach was proposed to simultaneously sonicate two or more foci with a single emission (Espindola et al. 2018). This has the advantage of retaining a high frame rate, yet achieving improved sensitivity to microbubbles. In the limit of one target, this beam reduces to a conventional focused transmission; and for an infinite number of targets, it converges to plane wave imaging. By interleaving targeting sequences with imaging sequences only volumes of tissue that contain visible bubbles can be targeted, thus reducing the number of transmit-receive events required to construct an image while improving the sensitivity. Experimental results have shown that the adaptive multi-focus sequence successfully detects 744 microbubble events at 60 mm, when they are undetectable by the plane wave sequence under the same imaging conditions. At a shallower depth of 44 mm, the adaptive multi-focus method detected 6.9 times more bubbles than plane wave imaging (1763vs. 257 bubble events).

Microbubble Isolation

One major area of research currently is the method by which these signals are separated. Indeed, for localization to work precisely, scattered signals from microbubbles need to be isolated in each individual image. Neighbouring echoes that are closer than the Rayleigh criterion in a single frame cannot be distinguished from one another and interference will make the corresponding localizations inaccurate. Notably, in (van Sloun et al. ICASSP, 2019), a deep neural network was specifically trained to learn the interference patterns of closely-spaced microbubbles and directly infer their locations. Yet, in most methods, these signals are typically rejected at this stage.

Ultrasound super-resolution has been demonstrated using relatively high microbubble concentrations in the blood, where signal separability is achieved by significantly limiting the number of microbubbles detected in each frame. As early as 2011, Siepman et al. and Couture et al. achieved microbubble isolation after bolus injection using pairwise frame subtraction, where preceding frames were subtracted to detect only the signals from moving or disrupted microbubbles in each image. In Siepman et al. this was performed using a high frequency of 50 MHz, meaning a considerably smaller diffraction-limited signal size, as well as the inherent compromise of limited penetration depth. With a lower frequency suitable for deeper imaging, Couture et al., and later Desailly et al. 2013, implemented pairwise frame subtraction, termed differential imaging (DI), using ultrafast imaging frame rates (up to 20,000 Hz for shallow tissue). This allowed microbubble isolation by detecting the decorrelation between successive ultrasound echoes due to the fast movement or disruption of bubbles. Although applicable, the disruption of microbubbles remains undesirable as it restricts microbubble tracking capabilities, as well as the visualization of microbubbles flowing through small microvessels.

The most direct way to achieve isolated signals is to reduce the concentration of microbubbles in the blood. Using an infusion, a constant microbubble concentration can be maintained that minimizes the likelihood of overlapping signals occurring. This method was implemented successfully in some of the earliest demonstrations of ultrasound super-resolution, including Viessman et al., 2013, O’Reilly and Hynynen, 2013, and Christensen-Jeffries et al. 2015, and has been used extensively in the field since. Notably, detection and isolation techniques are often intertwined, and some detection procedures may perform both of these steps to some degree. For example, either B-Mode, contrast-specific imaging techniques (e.g. PI or contrast pulse sequencing (CPS)) or SVD (Desailly et al. 2016) have been used primarily for detection, however, these also help to isolate signals since they reduce the number of bubbles which are visible in each frame. This can be due to, for example, some bubbles being off-resonance or weakly responding in the case of contrast-specific methods, or slow-moving or stationary bubbles in the case of SVD.

Despite the advantages of infusions for this technique, standard clinical contrast imaging protocols often involve a bolus injection. As such, soon after injection, the bubble concentration in the blood reaches its peak, and isolated detections are often near impossible (unless detection strategies are such that only a small proportion of bubbles are visualized). In these instances, the selection of suitable video segments during both the inflow and latter stages of the acquisition can improve the separability between individual microbubbles. This method was implemented in vivo by Ackermann et al., 2016. Alternatively, multiple lower-volume bolus injections can be performed to improve the duration of the desired bubble concentration as implemented in Errico et al. 2015 and Zhu et al. 2019. Additionally, destruction-replenishment sequences can be performed following contrast injection to capture the reperfusion of microbubbles into a region before their signals overlap, as in Opacic et al, 2018.

Although these methods aim to extract mainly isolated microbubble signals, an additional step is often required to reject any remaining interfering signals. For example, in O’Reilly et al. 2013, all in vitro frames containing too high sidelobe intensity (≥50% global maximum in frame) were rejected. Most other methods have involved searching for or rejecting signals based on their similarity to expected microbubble signatures. In Christensen-Jeffries et al. 2015, the size and intensity of any connected signal regions were compared to those of the expected PSF (point-spread function) based on characterization experiments. In Errico et al 2015, after spatio-temporal filtering procedures, a Gaussian model based on typical microbubble signal properties was used to perform a 2-D normalized cross-correlation with each interpolated frame. Similarly, in Ackermann et al, 2016, after background subtraction, a 2-D convolution of the foreground frame was performed with a Gaussian kernel to identify single microbubble signals for subsequent localization.

These methods can be performed with both conventional clinical imaging frame rates (≤ 100 Hz) (Viessman et al., 2013, O’Reilly and Hynynen, 2013, Christensen-Jeffries et al. 2015, Ackermann et al., 2016, Harput et al, 2018, Opacic et al, 2018), or at higher frame rates using plane-wave imaging (>100 Hz) (Errico et al 2015, Christensen-Jeffries et al, 2017b 3D UFFC, Lin et al 2017, Foiret et al, 2017, Song et al, 2017, Song et al, 2018, Zhu et al, 2019).

Localization

The prior knowledge that the diffraction-limited image of a microbubble originates from a single source allows one to estimate its location with a precision well beyond the diffraction limit. Within radio-frequency (RF) data, a single echo from a microbubble appears as a hyperbola, determined by the time-of-flight to arrive at each transducer element. In beamformed data, which are often more readily available, particularly in a clinical setting, microbubbles appear as individual ‘blurs’ in an image. Localizations from each domain are similar as they both relate to the fitting of the time-of-flight, and as distinct sources can both be determined very precisely. Nevertheless, the exact equivalence of the localizations in each domain is yet to be fully examined and may depend on factors such as the respective signal-to-noise ratio (SNR), the benefit of phase information in the RF space, and any nonlinear processes used to transfer into the beamformed space.

The localization step requires identification of some feature of the returned signal that corresponds to the bubble’s position. In most existing publications regarding ultrasound super-resolution, the position is expected to coincide with a measure related to the ‘center’ of the echo, due to the assumption that the image is a convolution of a point source with the system PSF. In early work using RF data, Couture et al. 2011 and Desailly et al, 2013 found a local axial maximum from the traveling hyperboloid in RF data lines by finding the summit of a parabola fit. In some of the earliest publications using post-beamformed data, localization was performed by calculating the intensity-weighted center of mass (Siepmann et al, 2011, Viessmann et al, 2013, Christensen-Jeffries et al, 2015), fitting a 2-D Gaussian function to either the original backscatter signal (O’Reilly and Hynynen, 2013), or after deconvolving with a predicted Gaussian PSF (Errico et al, 2015), or cross-correlation of signals with an expected response (Ackermann et al. 2016).

In optical localization microscopy, provided a sufficient number of emitted photons are sampled, the signal received from a molecule is well defined, and the intensity centroid is generally used to define its position to nanometer-scale. Likewise, if each microbubble from the insonated population behaves similarly, generating a signal close to the system PSF, then the aforementioned methods adequately represent the relative position of the contrast agents. However, more recently. Christensen-Jeffries et al. (2017b) demonstrated in vitro and in simulations that the considerable variability of bubble responses makes it challenging to predict both their linear and nonlinear response and that this variability should be taken into account during localization. It showed that centroiding, peak detection, and 2-D Gaussian fitting methods introduced hundreds of micrometers in error when using nonlinear imaging at low clinical imaging frequencies. As a result, a new ‘onset’ localization method was proposed to identify the beginning of the signal in the axial direction while ignoring ringing or elongated signal duration. This showed considerable improvement in localization error compared to existing methods and is particularly important in the resonant regime of microbubbles.

A recent article by Song et al. (2018) described the influence of the spatial sampling of the beamforming on the localization error. This study found that the Fourier analysis of an oversampled spatial profile of the microbubble signal could guide the choice of beamforming spatial sampling frequency. Furthermore, parametric Gaussian fitting and centroid-based localization on upsampled data had better localization performance and were less susceptible to quantization error than peak intensity-based localization methods. When spatial sampling resolution was low, parametric Gaussian fitting-based localization had the best performance in suppressing quantization error, and could produce acceptable microvessel imaging with no significant quantization artifacts.

The theoretical limits of ultrasound super-resolution, and under what conditions these could be achieved, is of vital importance. Many parameters, including physiological one, influences the separability of microvessels or other structures in super-resolution imaging. However, the best attainable resolution is limited by the localization precision of each microbubble. This lower bound is linked to the minimal time delay which can be estimated between similar echoes, known as the Cramer-Rao lower bound (Walker and Trahey, 1995). In 2015, Desailly et al. investigated the theoretical resolution limit to which a single, linear, point scatterer can be localized using hyperboloid fitting and validated this experimentally. In the far-field, Desailly et al. showed that in this situation the achievable resolution was dependent on the standard deviation of arrival time estimates on each channel RF line. These estimates depend on the pulse bandwidth, the pulse center frequency, the SNR, and the temporal jitter between electronic channels of the scanner. Both the number of transducer elements and the speed of sound also influence the resolution, while the lateral resolution was also dependent on the depth of the target and the size of the array aperture.

Alongside the extraction and isolation of individual microbubbles from tissue, the SNR of each microbubble signal is crucial to the performance of the localization algorithm. In Foiret et al 2017, the Coherence Factor (CF), the ratio of coherent intensity over incoherent intensity (Mallart & Fink, 1994), was applied to contrast pulse sequencing (CPS) images to improve the SNR of echoes from individual microbubbles. Ghosh et al. 2017 and Lin et al. 2017 have shown that selecting a population of larger microbubbles improves the SNR from individual microbubbles. Finally, due to the challenge of separating bubble signals from noise in images using just spatial information, Song et al. 2017 proposed to denoise the microbubble signal in a spatiotemporal domain using a nonlocal means filtering (NLM) filter to improve the separation of microbubble “tracks” from random background noise that does not resemble any feature or pattern. More robust denoising such as this is beneficial to improve the achievable isolation and localization precision.

Microbubble localization can also be improved by modifying the beamforming process, from a delay-and-sum to a maximum variance approach for example (Diamantis et al. 2018).

Tracking

Current ultrasound localization microscopy techniques usually go beyond the simple accumulation of microbubbles subwavelength positions to create an image. The fact that microbubbles remain exclusively intravascular allows the displacement of the microbubbles to be interpolated from the various localizations along a track. Both Errico et al. (2015) and Christensen-Jeffries (2015) used such an approach to establish the direction and the velocity of each microbubble at the micrometric scale. An important advantage of tracking is for the exclusion of artefactual microbubbles. Indeed, by removing very short tracks (a few images), only microbubbles with a coherent path are retained, forming an image with lower noise. Early-on, tracking algorithm were based on the closest-neighbor detector which identified from the localized position of a microbubble on image i the most probable localization on the image i+1. Algorithms have to define several thresholds such as the maximum distance to the next microbubbles for starting and ending a track.

Currently, more advanced algorithms can be exploited such as Monte Carlo based on Markov chain (Ackerman et al. 2015, Opacic et al. 2018), or joint tracking and detection in a Kalman framework that is regularized through optical flow estimates (Solomon et al., 2018). Hungarian assignment, already exploited in transportation analysis (Munkres 1957, Kuhn 1955), can also be used to improve tracking (Song et al. 2018,). New approaches to bypassing the localization process through a dynamical method have also been discussed (Alberti et al. 2018). Tracking was also performed in combination with motion correction which will be discussed in the next section (Hansen et al. 2016).

The images are often reconstructed by projecting the detected tracks on an image grid where the pixels are much smaller than the wavelength, 5×5 micrometers for example. However, it is important to note that the size of the vessel where a single microbubble is detected is misrepresented using this approach. For instance, a 5 micrometer “vessel” could simply be a larger vessel, such as an arteriole, where a single microbubble has passed. In fact, even the aorta would look like a capillary if only a single track is detected inside its lumen. The true size of a vessel should only be established with a large number of microbubble tracks, at a minimum sufficiently to cover the vessel diameter. Consequently, the minimal number of tracks required to reconstruct a vessel is the width of the vessel to be imaged divided by the super-resolved pixel size.

Motion correction

Super-resolution ultrasound imaging can achieve resolution down to 5 micrometers. However, motion in the body or from the ultrasound transducer can be several orders of magnitude beyond this level, especially since microscopic resolution requires long acquisition time (Hingot et al. 2019, Christensen-Jeffries et al., TUFFC 2019) — sometimes tens of minutes to reconstruct capillaries. A brute force approach is to fix the organ to be imaging with a stereotactic frame for the brain (Errico et al. 2015) or through respiratory gating (Christensen-Jeffries et al 2015). However, this approach is limited to organs not affected by breathing motion and its implementation is difficult to imagine in humans apart from neuroimaging. Lin et al. (2017) used respiratory gating to exclude images of tumors that were affected by breathing. Such an approach requires a steady motion of the organ which has to return exactly to its original position.

Several techniques are now being implemented to correct the motion based on the ultrasound images themselves. The phase-correlation of speckle motion can give an idea of the collective motion of tissue (Hingot et al. 2016). It is, however, necessary to remove the effect of the microbubbles which can bias the motion measurement. This can be done directly on B-mode has shown by Foiret et al (2017). More advanced non-rigid motion correction can also be implemented to take into account complex motions in the image (Harput et al. 2018). Indeed, because of its phase-sensitivity, IQ data provides ample information to detect micrometric motion as demonstrated in the fields of speckle-tracking (Walker et al. 1993) or transient elastography (Sandrin et al. 1999). Motion correction packages already implemented on Matlab can also be used directly on the IQ data.

However, since it cannot be tracked on the ultrasound data, out-of-plane motion cannot be easily corrected. Hence, motion correction is one of the principal arguments in favor of 3D ultrasound localization microscopy since displacements in all directions could be accounted for.

3D

Super-resolution performed on a 2D-plane will always be limited for several reasons. First, because tissue motion is inherently three-dimensional and its correction necessitate sampling in all directions, including the elevation axis. Second, because vessels follow tortuous paths in three-dimensions and a super-resolved imaging of small sections of numerous vessels intercepted by the imaging plane would give a restricted amount of information. Third, because microbubble position need to be accumulated for a long time, up to several minutes to attain the capillary level, reconstructing a volume through plane-by-plane imaging as in Errico et al. (2015), Lin et al. (2017) or Zhu et al. (2019) is difficult to transfer successfully to the clinic. Indeed, it could make ultrasound imaging even more user-dependent since the plane will have to be chosen and preserved for an extended amount of time, to avoid trading-off the underlying resolution.

Due to these 2D imaging drawbacks, super-resolution ultrasound imaging was rapidly implemented with matrix (Desailly et al. 2013) or hemispherical (O’Reilly et al. 2013) probes to perform ultrasound localization over volumes. Both these groups were able to map in 3D microbubbles flowing in microvessels with subwavelength resolution. However, these approaches were limited either in time resolution or in the reduced imaging volume. The latter issue also limited another approach exploiting two perpendicular probes for 3D reconstruction (Christensen-Jeffries et al. 2017).

Appropriate 3D imaging can be performed with matrix-array where each transducer can be addressed either through a beamformer in the probe or by connecting each channel to the ultrasound system. However, matrix arrays are costly and present some technological hurdles at higher frequencies, making the technology less available. Heiles et al. (2019) described in-vitro a 3D super-resolution imaging device where each of the 1024 (32×32) was connected on a programmable ultrafast scanner (Provost et al. 2014). In-vivo, this apparatus was able to reconstruct entire super-resolved volumes of the rat brain, alleviating the previously described drawbacks with 3D tracking and motion correction. Harput et al. (2019) reduced the number of connected channels by implementing super-resolution on a 2D sparse array. 3D super-resolution could even be performed with a clinical scanner as demonstrated by Christensen-Jeffries et al. (2019).

Super-resolution ultrasound is fast advancing toward volumetric imaging which could describe the microvasculature of entire organs within one microbubble bolus.

Trade-offs and limitations

With the potential for spatial resolution to be improved by orders of magnitude in ultrasound, the performance of the technique now depends on many new factors, including vascular flow rates, contrast-signal density, time-resolution, SNR, probe geometry, post-processing algorithms, and many more parameters. While each of the methodological steps has been investigated, adapted and improved by various groups, several new trade-offs have been discovered, and numerous strategies have been introduced to address them.

Firstly, much discussion has arisen in recent years regarding the requirement for sufficient measurement times to adequately visualize vascular structures. For all the aforementioned localization methods, microbubble signals need to be spatially isolated for accurate localization. Thus, there is an upper limit on the number of isolated signals attainable in each frame. The temporal domain is therefore used to gather sufficient localisations to adequately sample the structure of interest. Yet, since microbubbles travel within the bloodstream, slow flow rates associated with the smallest vessels limit our ability to sample these structures over short time-periods. Multiple groups have investigated this fundamental trade-off between acquisition time and visualization of the microvasculature, namely Dencks et al 2017, Dencks et al. 2019, Hingot et al, 2019, and Christensen-Jeffries et al. 2019.

Dencks et al. (2017) provided an expression for the acquisition time needed for the localization coverage in an image to saturate to a value assumed to be proportional to relative blood volume (rBV). The reliability of rBV estimates from shortened measurement times is then examined experimentally for bolus scans of mouse tumors at high frequency and conventional frame rates (40 MHz, 50 Hz), where approximately 45 – 170 s were found to be necessary for a 90% mapping of the microvessels. In Dencks et al. (2019) the saturation model was applied to clinical data of breast tumors (10 MHz, 15 Hz) and 28% – 50% filling were found for 90 s acquisition time. A study by Hingot et al. (2019) aimed to establish the link between imaging time and microvascular scale in super-resolved images and found that larger vessels (> 100 um) are fully reconstructed within 10 s, while mapping the entire capillary network would take tens of minutes. This is due to the limited blood flow in capillaries, along with their density in tissue. Moreover, the characterization of flow profiles was shown to be bound by even more stringent temporal limitations. Christensen-Jeffries et al. 2019 developed a generalized model to predict the optimal bubble signal density and acquisition time for user-defined imaging conditions, parameters and target vasculature. Similarly, this study estimated acquisition times of over 10 minutes for microvascular targets at 5 cm depth and λ/20 super-resolution. Both studies highlight that super-resolved vascular is inherently dependent on the physiology of the vasculature. Nevertheless, the distinct advantages provided by ultrasound super-resolution, along with the strengths of this technique over comparable imaging modalities, means several minutes of acquisition times may be acceptable in the clinical world.

Ultrasound super-resolution techniques have a particular advantage over high-frequency techniques to improve resolution without the conventional penetration trade-off associated with higher frequencies. Nevertheless, increased depth still has the capability to degrade the performance of ultrasound super-resolution. Firstly, imaging at greater depths will increase the minimum super-resolution acquisition time. This is because greater depths require lower transmit frequencies as in conventional imaging. For instance, ultrasound super-resolution imaging at great depth (>10 cm) or through the skull would require 1–3 MHz transducers, few centimeters depth would be achieved around 4–12 MHz, while small animal imaging would be performed at frequencies beyond 15MHz. Each time, super-resolution imaging would have an enhanced resolution with respect to conventional imaging, but it would still be degraded as the frequency is lowered to penetrate deeper. The resulting larger PSF size decreases the maximum number of separable signals that can be detected in each frame. Therefore, the acquisition time required to create a super-resolved image increases with depth, as shown in Christensen-Jeffries et al, 2019. This, along with the inherent time-of-flight vs depth relationship, therefore presents a trade-off in minimum possible super-resolution acquisition time with depth.

Another trade-off is with depth is reduced sensitivity, particularly using plane-wave imaging. Conventional focused imaging has a lower frame rate, but the well-established contrast and SNR advantages mean it has a higher sensitivity to microbubble detection than plane waves. For high frame rate sonications, the wide planar beam used to illuminate the field of view produces poor contrast and the transmitted energy is limited to minimize microbubble destruction over long imaging times. This means detecting microbubbles in microvessels at depth can be challenging. Strategies to improve the sensitivity of contrast agents at depth are being investigated for super-resolution imaging, for example, the use of larger microbubbles (Ghosh et al. 2017, Lin et al. 2017), and the use of adaptive multi-focus methods, which combine the high frame rates of plane wave imaging with the increased bubble detection sensitivity of focused beams (Espindola et al. 2018).

More generally, the performance of post-processing techniques and their influence on factors such as SNR and localization error presents potential trade-offs to be considered when making methodological choices. So far, Brown et al. 2019 have studied the performance of various microbubble detection methods using simulations, namely Pulse inversion (PI), differential imaging (DI), and singular value decomposition (SVD) filtering. It found that PI is the most appropriate for some applications, but also the most dependent on transducer bandwidth. PI was found to be largely independent of flow direction and speed compared to SVD and DI, so is more appropriate for visualizing the slowest flows and tortuous vasculature. SVD was shown to be preferable for high-frequency acquisitions, where localization precision on the order of a few microns is possible, but can introduce localization errors on the order of hundreds of microns over the speed range 0–2 mm/s, while DI is only suitable for flow rates >0.5 mm/s or as flow becomes more axial. Investigations such as these highlight the need for increased understanding of how different techniques can impact the super-resolution process.

Due to the nature of the research progression in this field, a number of other issues require increased attention in the near future. There is an inherent compromise between the SNR of the signal and the localization precision. This has been highlighted in studies such as Desailly et al, 2015, Ghosh et al. 2017, Lin et al. 2017 and Espindola et al. 2018. Methods to understand and improve the SNR particularly with plane wave imaging require further attention. Exploration of the impact of SNR, specifically in relation to depth and aberration effects is greatly needed. Additionally, with increased acquisition time, and with the field moving increasingly into the 3D domain, managing and analyzing large data will become more and more important in coming years. However, it has to be stated that ultrasound localization is a powerful data-compression technique. Indeed, radio-frequency channel data —terabits in size for long 3D acquisitions— can be converted to an accumulation of hundreds of thousands of small vectors —a few bits each— representing the super-resolved path of microbubbles.

Another important trade-off in ultrasound super localisation based method is the concentration of the injected microbubbles. Too low a concentration would certainly further prolong the acquisition time, but increasing the concentration would increase the probability of having more than one microbubbles in the same imaging resolution cell, and too high a concentration would cause many detected signals to be not useful for super-localisation. Such multiple bubble events, if not rejected, would contribute to the noise of the super resolution image. Christensen-Jeffries et al. 2019 has demonstrated that an optimal bubble concentration exists for super-resolution ultrasound imaging.

Super-resolution ultrasound imaging needs to demonstrate that it can provide superior microvascular and microstructural information for clinical applications to be relevant. In particular, it should be compared to the clinical diagnostic benefit of perfusion ultrasound imaging. Is there a benefit to display super-resolved microbubble tracks as compared to intensity images or maximum intensity projection? For instance, if we reconstruct a single microbubble track, it is impossible to determine if the microbubble was moving inside a large or a small vessel. Several microbubbles are hence required in each vessel to determine its true size and bifurcation. To assess its accuracy for larger vessels, super-resolution ultrasound imaging should be compared to other vascular imaging modalities, such as ultrafast Doppler (Bercoff et al. 2011) or optical techniques such as microscopy. The field of ultrasound should explore these concepts of accuracy more deeply and move beyond the concerns over the smallest observable vessel or the distinction between two vessels.

Ultrasound super-resolution using phase-shift agents

A fundamental difference between the localization-based optical super-resolution methods and the microbubble based ultrasound counterparts is that there is no switching on and off of the contrast agents for the ultrasound approach. In optical super-resolution such as PALM (Betzig2006) and STORM (Rust2006), some individual contrast agents (i.e. fluorescence molecules) from a high concentration population are stochastically switched on and off, creating “blinking” signals based on which these individual agents can be localized. The accumulation of the localizations depends on the stochastic switching on and off of different sub-populations of the agents. On the other hand microbubbles, once injected, are already in an “on” state. Therefore, for localization-based ultrasound super resolution approaches to work, low concentration of the bubbles are required to ensure the spatial distribution of the bubbles are sparse, and physiological flow during imaging is also required to displace the bubbles so that different localizations can be accumulated over space and time.

While localizing the same contrast agent over time in the presence of flow offers the opportunity to generate super-resolved velocity map of micro-flow (Errico et al. 2015, Christensen-Jeffries 2015), a limitation of the approach is that they heavily depend on the agent concentration and presence of flow, and consequently may take much longer acquisition time. Regardless of the imaging frame rate, time has to be spent waiting for spatially isolated individual bubbles to flow pass the microvessels of interest (Hingot 2019, Christensen-Jeffries 2019). Furthermore, the in vivo microbubble concentration is difficult to control and it can change quickly over time, which can cause either too few localizations in small micro-vessels, or in larger vessels with high agent concentrations, too many incorrect localizations of multiple bubbles which if not discarded can contribute to image noise.

Phase change nanodroplets for ultrasound imaging and therapy:

Phase change droplets, typically made from perfluorocarbons, can change phase from liquid to gas (i.e. be vaporized into bubbles) by ultrasound (Kripfgans 2000, Shpak2014). Such phase change agents have been studied in the past couple of decades for both therapeutic and diagnostic applications (Kripfgans 2000, Kawabata et al. 2005, Rapoport 2009, Reznik2011 and Sheeran 2012) and recently has been shown to offer subwavelength ultrasound drug-delivery and super-resolution imaging (Hingot2018). Most research in phase-change agents are focused on nanometric agents, which allow longer circulation, tumor deposition through the enhanced permeability and retention effect, while reducing the size of the resulting microbubbles.

As alternative ultrasound imaging contrast agents, nanodroplets offer three potential advantages over microbubbles. First, the nanodroplets can be selectively activated, both spatially and temporally, to provide an ultrasound contrast signal on demand. Second, the nano-size of nanodroplets potentially allows extravasation, e.g. into cancerous tissue due to its leaky vasculature and enhanced permeability and retention effects, offering contrast beyond vasculature. Third, it has been found that nanodroplets can last longer during in vivo circulation than microbubbles (Sheeran2015). In the past decade it has been well demonstrated that the vaporization of such phase change nanodroplets into microbubbles can be controlled by acoustic pressure (Sheeran 2013), and it is also affected by their own characteristics (e.g. size distribution and boiling point of the gas) as well as external factors (e.g. microvessel confinement (Lin 2018) and attachment to cells (Zhang UMB 2019). Furthermore, it has also been well demonstrated that microbubbles can be disrupted/switched off by ultrasound (Chomas 2001).

Acoustic wave sparsely activation and localization microscopy (AWSALM):

The capability of phase change nanodroplets to be switched on and off by ultrasound on demand allows ultrasound super-resolution imaging to mimic STORM or PALM in optical super-resolution. The switching-off mechanism can be either the recondensation of the resulting microbubble back into a nano-agent or its disappearance through disruption or dissolution. This has recently been demonstrated by Zhang et al (Zhang 2018 APL) through acoustic wave sparsely activated localization microscopy (AWSALM) in two proof-of-concept studies (Zhang 2018 APL, and Zhang 2019 UFFC). One of the fundamental distinctions between bubble- and droplet-based super-resolution imaging techniques is that, with bubble based super-resolution approaches, accumulated signals are limited by low bubble concentration and depend on flow decorrelation, which is slow in the smaller vessels. However, with the droplets being activated and deactivated at the ultrasound pulse repetition frequency, signals can be accumulated as quickly as the imaging pulses are sent, being able to achieve much faster super-resolution imaging. In droplet-based ultrasound super-resolution approach, much higher concentrations droplets can be used to pre-populate the vessels (as droplets are invisible to ultrasound before activation) and localizations can then be accumulated quickly by switching them on and off even with very slow or no flow.

In such approaches there are two key factors; droplet composition and the transmitted ultrasound wave parameters. The transmitted ultrasound amplitude has to match the vaporization threshold of the droplets to activate them, and the threshold depends on the boiling point of the perflurocarbon gas, their size distribution and whether the droplets are encapsulated by a shell. In its initial realization, in Zhang et al. (2018), shelled decafluorobutane (boiling point −1.7 °C) nanodroplets were used, focused waves with relatively high ultrasound amplitude (MI=1.3) were swept through the imaging area to switch on new droplets while switching off previously activated droplets at the same time, followed by lower amplitude (MI=0.25) imaging plane waves. Such an approach of separate activating and imaging pulses is necessary in this case as the activation of the decafluorobutane droplets requires acoustic pressure that is difficult for plane-wave to achieve. Consequently, a significant amount of time is spent on sweeping the focused activating pulses and this slows down the imaging.

More recently the same authors have developed a fast version of the AWSALM (Zhang UFFC 2019, figure 2), where shelled octafluoropropane (boiling point −37 o) nanodroplets are used which can be activated by ultrasound with a relatively low MI that plane wave transmission can achieve. The study has demonstrated that octafluoruopropane (OFP) nanodroplets can be simultaneously imaged, activated and deactivated at very high frame-rate (5000 fps) at an intermediate acoustic amplitude (MI=0.76) to perform ultrasound super-resolution imaging. Two tubes, which contain high concentration droplets without flow and are separated by a distance below the ultrasound diffraction limit, are well resolved within 200 milliseconds, achieving a two orders of magnitude improvement in temporal resolution when comparing to existing ultrasound super-localization techniques.

Figure 2:

Fast ultrasound super-resolution imaging achieved by fast-AWSALM (a) Conventional B-mode image (b) SVD-filtered image (c) super-resolution image by a 200 milisecond data acquisition (d) optical image of the 200 μm cross-tube phantom. (e) and (f) show the resolution measurements at different lateral ROIs indicated by the red lines on the images. (Figure from Zhang et al. UFFC 2019)

Photoacoustic super-resolution:

Photoacoustic super-resolution has been developed taking advantage of the nonlinear photoacoustic signal dependence of certain molecules or nano-particles on the laser energy (Nedosekin2014, Danielli 2014). For the perforocarbon nanodroplets, instead of using ultrasound they can also undergo phase change under laser excitation, allowing photoacoustic super-resolution. High-boiling-point phase change nanodroplets for super-resolution photoacoustic imaging (Luke 2016) have been demonstrated. A unique feature of this approach is that such droplets can condense back to droplets after being vaporized into bubbles under relatively high intensity laser pulses, creating “blinking” signals for super-localization. An advantage of this approach is that the same droplets can be repeatedly used. Flowing optically absorbing particles have also been used for photoacoustic super-resolution (Vilov2017, Luís Dean-Ben 2018). Comparing to acoustic activation, such photoacoustic techniques use lasers and the penetration depth is restricted to regions that can be illuminated optically (typically up to a few centimetres). Furthermore, as the pulse repetition frequency of photoacoustic laser was usually low (10 Hz), longer acquisition time is required.

Furthermore, due to the size of the nano-droplets, they may be able to leak out of the vasculature and it is possible to offer in vivo super-resolution beyond the vascular space, e.g. to visualize greater details of tumor tissue structure by using targeted nanodroplets.

Ultrasound super-resolution without localization

Standard localization-based techniques for ULM rely on sufficient separability of microbubbles, placing an upper bound on the allowable concentration. Depending on the desired image fidelity, this in turn places a substantial lower bound on the acquisition time, typically being on the order of minutes (Opacic, 2018), (Hingot, 2019), (Dencks, 2019). By introducing other trade-offs, ultrasound super-resolution can be performed faster through different strategies exploiting structured illumination (Ilovitsch et al. 2018) or the signal structure.

In this section, several non-localization-based techniques are described, which intend to push this lower bound down significantly by permitting higher concentrations without compromising precision. These methods make effective use of structural signal priors to overcome the limitations of localization-based techniques for scenarios with overlapping point spread functions (PSFs). Such priors are obtained explicitly through signal models or implicitly by learning from data. An overview of these approaches is given in Figure 3.

Figure 3:

An overview of several non-localization methods that leverage priors through signal structure or by learning from data. An input CEUS sequence (a) can be modelled to derive estimators that exploit signal structure (b), or to generate realistic data to train data-driven estimators in the form of deep neural networks (c).

Exploiting signal structure

We first assume that the tissue clutter has been removed. This can be achieved by using high-pass filtering, singular value decomposition, or deep learning methods (Cohen et al, 2019). The image obtained can then be written as a convolution between the PSF of the system and the microbubbles. We can express this relation in matrix-vector form by vectorising the low-resolution image frame into a vector y As shown in (Bar-Zion et al., 2017), (Bar-Zion et al., 2018) the resulting vector follows the measurement model:

$$y=Ax+n,$$ (1)

where A is the PSF matrix consisting of shifts of the PSF which relates the vectorized microbubble distribution on a high-resolution grid x to the image, and n is a noise vector. The goal is to recover the high-resolution vectorized image x.

In its most basic form, signal structure can be exploited by realizing that the microbubble distribution within a frame x is highly sparse on such a high-resolution grid (van Sloun et al., 2017). This implies that the vector x contains only a few non-zero values and can therefore be recovered by relying on the compressed-sensing literature (Eldar, 2012, Cambridge University Press), which provides methods for recovering sparse vectors from under-determined linear systems such as (1) using computationally efficient methods. In particular, one can formulate a convex minimization problem that balances a data fidelity term and a structural sparsity promoting term, which results in sparse solutions to the problem of (1) on a high resolution grid.

To allow for increased overlap between microbubbles, one can extend beyond single frames by processing multiple frames of the form (1), leading to what is known as a multiple measurement vector model (MMV) (Eldar, 2009, IEEE trans. on Information Theory). Such models permit exploiting signal structure not only within a frame but also across frames, by relying on the prior that the non-zero elements in each of the frames are located in the same positions. In line with this, the authors in (Bar-Zion et al, 2017) propose to compute high-order statistics of the temporal microbubble signals at each pixel, achieving high-resolution imaging in a fashion that is similar to super-resolution optical fluctuation imaging (SOFI) in fluorescence microscopy (Dertinger, 2009, National Academy of Sciences). This ultrasound variant of SOFI makes effective use of the statistical independence between the fluctuations of CEUS signals originating from different vessels. The attainable resolution gain was shown to scale with the order of the statistics. In practice, the authors propose to use second order moments, since estimation of high-order moments requires an exponentially increasing number of image frames to retain the same SNR level.

Sparsity-based Ultrasound Super-resolution Hemodynamic Imaging (SUSHI) extends upon this SOFI-inspired solution by also exploiting sparsity of the underlying vascular architecture, thereby further improving spatial resolution (Bar-Zion et al., 2018). This prior can be incorporated by solving a minimization problem in the correlation domain with a sparsity-promoting regularization term on the high resolution correlation image, thus accounting for both the joint sparsity and the statistical properties of the image. Other than exploiting sparsity of the vasculature itself, sparse representations may also be used in a transformed domain, e.g. through a wavelet or total variation transformation. SUSHI was shown to effectively exploit the distinct temporal fluctuations of microbubbles in different vessels when imaging at a sufficiently high frame rate, i.e. when leveraging ultrafast imaging. In this scenario, spatial resolutions beyond the diffraction limit were attained at an unprecedented temporal resolution of 25Hz.

When frame-rates are low compared to the vascular flow velocities, the differences between temporal fluctuations among vessels and their high-order statistics exploited in CEUS-SOFI and SUSHI are less significant. To address this, another method that also resides within the MMV framework, simultaneous sparsity-based super-resolution and tracking (triple-SAT) has recently been proposed (Solomon et al., 2018). Triple-SAT combines frame-by-frame weighted sparse recovery for detection and simultaneous tracking of the microbubbles, thereby explicitly exploiting flow as a prior in the detection process. In doing so it leverages both a structural spatial sparsity prior as well as a flow-based prior to resolve vascular architectures with super-resolution. While the tracking and data association methods used in Triple-SAT share similarities with those presented in (Ackermann et al., 2016), the method distinguishes itself by the incorporation of the aforementioned tracking-based flow prior in the detection stage, in conjunction with crude optical-flow-based velocity estimates within a Kalman framework.

From an optimization point of view, the methods presented above can all be cast in a similar form, as solving regularized least squares problems of the form:

$$\widehat{x}=arg\underset{x}{max}\left\{\Vert Ax-y{\Vert }_2+\lambda R\left(x\right)\right\},$$ (2)

where the first term represents a data fidelity term, and R(x) denotes a regularization term that embeds the structural priors. The fidelity term may represent the image directly as in (1), or may represent the image correlation function, as in SUSHI and SOFI. In those settings, x effectively represents the variance of the pixels in the underlying high-resolution image. By taking into account the structure of A, problem (2) may be solved in the Fourier domain, leading to computationally efficient methods (Solomon et al., 2019, SIAM).

In the case of frame-by-frame sparse recovery, one may choose R(x) = ‖x‖₁, promoting sparse solutions. For Triple-SAT, R(x) = ‖diag(w)·x‖₁, with w being a flow-prior-based weighting vector, promoting solutions that are both sparse and close to the tracking predictions. In SUSHI, R(x) = ‖Ψx‖₁, with Ψ denoting a basis transformation that maps to a domain in which the vascular architecture is sparse (e.g. through the wavelet transform) and x represents the image variance. In (van Sloun et al., 2017) and (Bar-Zion et al., 2018), (2) was solved through an efficient Fourier-domain implementation of the fast iterative shrinkage and thresholding algorithm (FISTA).

Leveraging deep learning

Exploiting structural signal priors in some domain, either at a single frame level or in a MMV model, has provided an effective set of tools that can alleviate the harsh constraints on allowable microbubble concentration. However, the potential it carries for reducing the acquisition time comes at the cost of high reconstruction time; even highly-optimized Fourier-domain implementations of FISTA rely on a time-consuming iterative procedure. Appropriate optimization settings and the optimal regularization (thresholding) parameter λ moreover depend on the actual sparsity of the data, which in practice varies within and across acquisitions. Furthermore, exact knowledge of the PSF is needed to form the measurement matrix.

The above challenges motivate the development of a solution that is fast, has fixed complexity, and is robust across a wide variety of imaging conditions. To that end, the authors in (van Sloun et al., 2018, Arxiv) and (van Sloun et al., 2019, ICASSP) propose Deep-ULM, a method that leverages recent advances in deep learning to fulfill these desirable properties. Deep-ULM uses a deep fully-convolutional encoder-decoder neural network to map low-resolution input frames to sparse localizations on a high-resolution grid. It is trained using simulations that reflect a wide variety of imaging conditions, covering varying microbubble concentrations, backscatter amplitudes, background clutter and noise, and variations in the PSF. Once trained, inference by Deep-ULM is very fast, recovering more than a thousand high-resolution patches of 128×128 pixels per second using GPU acceleration. As such, it is about four orders of magnitude faster than the iterative FISTA scheme. Beyond speed-up, Deep-ULM also yields improved performance (in particular for high concentrations), which may be attributed to its ability to learn the relation between specific interference patterns of ultrasound waves reflecting off closely-spaced microbubbles, and their locations (van Sloun et al. 2019, ICASSP).

Developing applications

The previous section mostly discussed technical aspects of super-resolution techniques. We focused on ultrasound localization microscopy as it is currently the most common approach for ultrasound super-resolution. However, the choice of the appropriate technique depends mostly on the requested spatial resolution and temporal resolution. ULM still achieves the highest resolution, but an acquisition with sufficient microbubble track takes an extended time. Fluctuation-based approaches or structured illumination could provide an interesting compromise between resolution and frame-rate — a middle-way between ultrafast Doppler and ULM.

We will now present two main fields of applications which could benefit strongly from ultrasound super-resolution: oncology and neurology.

Oncology

Medical imaging is widely recognized as one of the most significant advances in the war on cancer. Early detection of lesions through screening, or diagnosis of suspicious lesions found by other screening methods while tumors are still small and before metastasis drastically increases the chances of successful treatment (WHO, Cancer Detection). Imaging during treatment provides crucial information regarding efficacy of the therapeutic approach. Due to its low cost, widespread accessibility, and safety, ultrasound has the potential to be a preferred modality for cancer imaging. Yet, traditional grayscale ultrasound falls behind other modalities for certain cancer imaging applications. Breast imaging is one such application. Breast cancer is the most common cancer in women and second most common cancer overall (WCRF). Mammography is the first choice for screening for breast cancer, with sensitivity typically reported between 70%–90% (albeit reported with varying ranges) (Heijblom et al. 2011). Ultrasound can be combined with mammography to improve this sensitivity, although the specificity of this combination is poor; as low as 40% in women with dense breast tissue (Heijblom et al. 2011). Mammography is even more challenged in women with saline or silicone implants, which drastically attenuate x-rays and substantially limit imaging efficacy. The result is that many imaging results are indeterminate and biopsy is recommended for pathological confirmation. Unfortunately, the biopsy process is associated with emotional stress, risk of complications, and additional time and cost (Rey et al 2005, Montgomery et al. 2010). Improved ultrasound imaging methods which provide improved specificity and sensitivity to cancer would provide clear advantages in screening and diagnosis.

These abnormalities provide an opportunity however for a diagnostic imaging technique which can visualize microvascular features.

Although traditional grayscale or Doppler ultrasound techniques are challenged to visualize microvascular features on the scale of early tumor angiogenesis, it has been demonstrated that the biomarker of angiogenesis can be detected with the use of high-resolution contrast ultrasound. Recently, investigators have demonstrated visualization of cancer associated microvascular abnormalities through acoustic angiography, a high-resolution superharmonic imaging technique (Gessner et al. 2012). Acoustic angiography, which requires dual-frequency transducers in order to excite ultrasound contrast agents near resonance and receive higher order harmonics, can provide 3-D maps of microvasculature in shallow tissues with a resolution on the order of 150 microns at 2 cm (Gessner et al 2010). Using this microvascular imaging method, investigators have observed clear indicators of cancer-associated angiogenesis in both xenograft and spontaneous tumors (Figure 4) (Shelton et al. 2015, Rao et al. 2015), and preliminary data suggests angiogenesis as a biomarker can improve the sensitivity and specificity of ultrasound to cancer substantially over grayscale imaging. Acoustic angiography is fundamentally limited by the diffraction limit, however, and hence resolution is ultimately limited by the frequency of the imaging system with its inherent tradeoff in frequency-dependent depth attenuation, which will prohibit acoustic angiography from ever being a technique which can resolve small vessels involved in angiogenesis in most tissues deeper than a few cm.

Figure 4.

Application of microvascular imaging to assess the angiogenic biomarker of cancer: Maximum intensity projections of 3-D acoustic angiography imaging of microvasculature for healthy (control) tissue and tissue surrounding fibrosarcoma tumors in a rat model. Dotted circles indicate approximate location of solid tumor mass, and field of view is approximately 2.5×2cm. Reproduced with permission from R. C. Gessner, S. R. Aylward, and P. A. Dayton, “Mapping microvasculature with acoustic angiography yields quantifiable differences between healthy and tumor-bearing tissue volumes in a rodent model,” Radiology, vol. 264, no. 3, pp. 733-40, Sep 2012 ©RSNA.

Super-resolution imaging, using ultrasound localization microscopy, however, is not bound to diffraction limited resolution, making this technique a potentially powerful tool to identify malignancies via their angiogenic fingerprint even in deep tissues. Lin et al. applied super-resolution imaging to visualize 3-D microvascular patterns in fibrosarcoma tumors in rats, and compared microvascular features to control healthy tissue (Figure 5) (Lin et al 2017). Similar to prior data acquired with acoustic angiography, super-resolution imaging demonstrated that tumors exhibited characteristic signatures of angiogenesis such as a high degree of tortuosity. Although these early studies were performed in rodents at shallow depths, data was encouraging that this technique would readily translate to deeper tissues in humans as well. The primary challenge with microvascular imaging analysis is that it requires 3-D volume imaging to be most effective, and motion correction can easily corrupt the accurate mapping of microvessels on the order of tens of microns.

Figure 5.

Maximum intensity projections of 3D super-resolution imaging of healthy microvasculature (a-b) and tumor associated microvasculature (c-d). Smallest vessels resolved were approximately 25 microns in diameter, approximately 6x improved from achievable with acoustic angiography at a similar depth. Reproduce with permission from: F. Lin, S. E. Shelton, D. Espindola, J. D. Rojas, G. Pinton, and P. A. Dayton, “3-D Ultrasound Localization Microscopy for Identifying Microvascular Morphology Features of Tumor Angiogenesis at a Resolution Beyond the Diffraction Limit of Conventional Ultrasound,” Theranostics, vol. 7, no. 1, pp. 196-204, 2017.

Lymphatic system, including lymph nodes and vessels, plays a crucial role in oncology. In patients with cancer, tumor cells often traffic through lymphatic channels to regional lymph nodes and beyond to distant sites. Accurately identifying and quantifying metastatic lymph nodes remains essential for prognosis and treatment planning in cancer patients (Grills et al. 2003). Recently super-resolution of rabbit popliteal lymph nodes micro-vasculature has been demonstrated (Zhu J. et al., 2019, Fig 6). Some technical aspects of imaging the slow lymph flow in lymphatic vessels in order to track the sentinel lymph nodes are also studied (Zhu J et al., UMB in press).

Figure 6:

2D projection of popliteal lymph node through 17 super-resolution image slices covering 1.7 mm thickness: (left) Depth color-coded super-resolution maximum intensity projection (MIP) images, where hue encodes the image slice with the maximum intensity and saturation represents the number of microbubbles localized at that depth. (right) Velocity color-coded MIP, where color shows velocity and direction. Top and bottom groups of images obtained from 2 different lymph nodes. (Adapted from Zhu J. et al., Radiology 2019)

A particular challenge for super-resolved imaging of the tumor vasculature is the complexity of the vessel morphology with a less organized and sometimes immature vessel tree. Adding to this is the higher slice thickness in clinical imaging that projects more intersecting tracks into a single image and the lower framerates that make the association to tracks more ambiguous. To resolve this, tracking algorithms incorporating linear motion models based on Kalman filters have been proposed (Ackermann et al. 2016) . These methods were applied to preclinical tumor xenografts for successfully differentiating tumor types in a radiomics analysis based on quantitative parameters of vessel morphology (Opacic et al. 2018). Additionally, clinical feasibility of the method was demonstrated for chemotherapy monitoring of breast cancer (Dencks et al., 2019; Opacic et al., 2018). An example for microbubble tracking analysis of a contrast enhanced ultrasound scan of a patient with breast cancer is shown in Figure 7. Here, the vascular architecture is displayed in detail, which consist of a highly vascularized rim from which vascular assemblies spoke-like proceed towards the tumor center. These are connected with vascular hotspots that show high blood velocities and distribute the blood to the overall tumor tissue.

Figure 7:

B-mode image (A) of a breast cancer patient showing a hypointense, irregular lesion with unsharp margins. The MIP image after microbubble injection (B) confirms the lesion to be highly vascularized but details in the vascular architecture can hardly be captured. Profound information on the vascularization is available from the microbubble tracking analysis illustrating microbubble tracks (C), the directions of blood flow (D) and individual vessels’ velocities (E).

However, there are still several challenges to realize microbubble tracking analyses in clinical routine. The different human organs have distinct functional and anatomical vascular properties that need to be considered to avoid an overload with microbubbles that makes it impossible to assign individual tracks. Furthermore, the resolution in clinical ultrasound scanners is usually lower than in systems used for small animal imaging since lower frequencies need to be applied to sufficiently penetrate the tissue. Due to the larger voxel sizes and particularly the thicker image slices, it is more difficult to calculate the correct position of the microbubbles and voxels may even contain several microbubbles at the same time both making it more difficult to determine correct microbubble tracks. Thus, CEUS protocols and microbubble doses need to be carefully adapted, and the consideration of pre-knowledge on the vascular anatomy or the application of deep learning algorithms may help to correctly assign tracks. Another challenge for clinical translation is organ motion. This is less critical for superficial lesions in the skin or the muscle as well as for the brain but becomes highly relevant for abdominal organs such as liver, kidney, spleen and pancreas. Motion correction in the image plane was implemented (Dencks et al., 2019; Harput et al., 2018; Hingot, Errico, Tanter, & Couture, 2017) and needs to be further refined in future studies. However, motion that occurs orthogonal to the image plane remains a challenge. In this context, advancing CEUS imaging from 2D to true 3D using matrix transducers, which was demonstrated already in vitro (Harput et al., 2019; Heiles et al., 2019), may not only allow to display the vascular network in much greater comprehensiveness and detail but also holds promise for a precise motion correction in all directions (Harput et al., 2019).

In conclusion, clinical application of microbubble tracking analyses is feasible in cancer application, provides encouraging preliminary data and some vendors of ultrasound devices also consider implementing such analyses methods in their scanners’ software. In addition, in agreement with research on other noninvasive imaging modalities, there is high current interest in the extraction of multiple quantitative image features and their radiomic analysis to receive a disease-specific signature. Recently, it was shown that the radiomic analyses of contrast-enhanced ultrasound data reliably allows to distinguish different tumor types in mice (Theek et al., 2018). In this context, approaches on microbubble localization and tracking have several advantages over the established methods that base on intensity time curves: 1. the imaging protocols are considerably more simple and robust, 2. no complex pharmacokinetic models need to be applied that base on assumptions that may not be fulfilled (e.g. stable blood concentration of microbubbles or one sole feeding vessel), and 3. significantly more functional parameters can be extracted than by other contrast-enhanced ultrasound (CEUS) analysis methods since individual tracks and vessels are studied. However, there is still significant room for further research on hardware, algorithms, software, and scan protocols to render the methods robust and reliable enough to improve the diagnostic precision in clinical routine imaging of cancer.

Neurology

The ability to resolve small vessels deep in the brain has implications in neurology for both diagnostics and therapeutics. Small vessel pathology is increasingly recognized as a contributor in disorders such as stroke and Alzheimer’s disease [Pantoni 2010, Gorelick 2011, Wardlaw 2013], while tumors are characterized by irregular, tortuous vasculature. Anatomical and functional imaging on a capillary level scale could provide powerful new insight into the workings of the healthy and pathologic brain.

The skull bone stands as the major hurdle to high resolution ultrasound brain imaging. Bone is highly attenuating and aberrating to sound, particularly at higher frequencies [Fry 1978, Pichardo 2011]. Thus ultrasound imaging and therapy must either be conducted at lower frequencies that better penetrate bone or must be performed through acoustic windows, such as the temporal or suboccipital windows [Lindsay 2011, Lindsay 2013]. O’Reilly and Hynynen [O’Reilly 2013] used a hemispherical array designed for transcranial ultrasound therapy to image single bubbles in a tube phantom through an ex vivo human skull cap, using the emissions to non-invasively correct image aberration and fitting the expected PSF to generate a super-resolution image (Fig 8a). This proof-of-concept study demonstrated the ability to imaging through the intact skull bone, with the potential to image a large volume of the brain with a spatial resolution <50 um. However, the current approach suffers from long acquisition times due to the low bubble concentration used and the fact that the focal volume required steering to generate the larger image. Employing larger and/or multiple excitation volumes, as well as simultaneously localization of multiple bubble through fitting or deconvolution [Foroozan 2018] could reduce total acquisition time.

Figure 8 –

From left to right: Micro CT image of a spiral tube phantom with nominal internal diameter of 255 μm; Super-resolution ultrasound image of the phantom through a human skull cap using a hemispherical array (transmit 306 kHz, receive 612 kHz); Merged image showing agreement between ultrasound and microCT. (reproduced from O’Reilly 2013 by permissions of Wiley company, all rights reserved)

In contrast, in their landmark publication Errico et al. [Errico 2015] produced stunning in vivo images of rat brain vasculature at 15 MHz (Fig 9), both through craniotomy windows and through thinned skull bone. While not performed at a frequency amenable to penetrating the human skull bone, this approach could be applied in its current form to image the human brain intraoperatively or in premature infants, where the anterior and posterior fontanelles are still open, with exciting potential. This study leveraged ultrafast plane wave imaging and the movement of bubbles between frames to obtain isolated bubbles without having to use very dilute concentrations. Errico et al., also noted that in theory a reduction of their imaging frequency to 2.5 MHz could still yield 6 um isotropic resolution. At this frequency, imaging though the thin temporal bone of the human skull could be possible. In fact, multiple matrix arrays imaging simultaneously through the two temporal bones have been used by Lindsay et al., [Lindsay 2011, Lindsay et al 2013] to produce three dimensional images of the Circle of Willis in human volunteers at normal resolution. Additionally, Soulioti et al have recently demonstrated transcranial super-resolution imaging of a tube phantom through human temporal bone with a 2.5 MHz diagnostic transducer [Soulioti 2019], achieving similar accuracy to [O’Reilly 2013], further supporting the feasibility of this approach. The implementation of super-resolution methods on such a system could create a powerful tool for imaging stroke.

Figure 9 –

Super-resolution image (15 MHz) of rat brain vasculature through thinned skull bone using an ultrafast scanner at 500 frames per second. 150 seconds of acquisition were required to localize and track approximatively 1 million separable sources per hemisphere (From Errico 2015, Nature PG all rights reserved).

A perhaps non-intuitive challenge to be faced in brain imaging, beyond distortion and signal loss due to the skullbone, is that of organ motion. Human brain motion as a result of the cardiac cycle has been measured to be as high as 0.5 mm, and varies by structure [Enzmann 1992]. Breathing motion may cause additional lower frequency shifts in the echo locations. Such displacements are significant relative the structures that super resolution ultrasound aims to resolve. Motion correction algorithms [Hingot 2017] will thus be necessary to image smaller vessels in the brain.

Beyond diagnostic imaging the development of super resolution ultrasound methods will open up great potential for guiding and monitoring image guided interventions in the brain, such as ultrasound-mediated opening of the blood brain barrier [Hynynen 2001, McDannold2006, Capentier2016; Lipsman2018; Mainprize2019] or sonothrombolysis [Culp 2011; Pajek2014].

Super resolution imaging thus has immense potential to drastically alter our understanding of the brain and our ability to treat it.

Outstanding challenges and perspectives

As we have seen, ultrasound super-resolution imaging can explore the animal and human vasculature in-vivo and in-depth at a scale unattainable until recently. Both oncological and neurological applications have been discussed here, leaving many potential others, such as kidney imaging, which has been explored by several authors (Song et al. 2017, Foiret et al. 2017) or lower-limb imaging which opens the field of microvascular imaging in diabetic patients (Harput et al. 2018).

Despite its numerous advantages, super-resolution ultrasound imaging is still limited by several aspects: acquisition time, signal-to-noise ratio, dependence on exogeneous contrast agents, motion, data overdose, lack of a gold-standard, partial reconstruction, exploitation of ultrafast scanners uncommon in the clinic, restriction to the vascular network, lack of 3D reconstruction, etc.

One of the important challenge to tackle in the field of super-resolution ultrasound imaging is the determination of its accuracy. In depth and in-vivo, no other imaging modality can provide a similar resolution for vascular imaging. Hence, it is difficult to determine if the observed vessels are appropriately reconstructed, especially their diameter and velocimetry. One approach is to observe the same plane of a shallow microvascular network with ULM and optical means with an orthogonal setup (Christensen-Jeffries et al. 2015). However, further studies would be needed, potentially with the help of confocal microscope, multi-photon microscopy or optical coherence tomography to assess the accuracy of ULM at least within the first mm. For in-depth organs, ex-vivo 3D tissue clearing (Renier et al. 2014) could give further confirmation.

Long imaging time can be partly alleviated through deep-learning, but further development are required in ULM to separate closer microbubbles so that injected concentrations could be increased. More localization per frame would mean faster and more resolved images. This is especially true when conventional scanners are used in the clinic.

Vast improvement in signal-to-noise and contrast-to-tissue ratio were achieved in the last 30 years for perfusion imaging with contrast agents. Specific sequences were developed, but new ones are probably required specifically to extract individual microbubbles. Any gain in SNR and CTR improves drastically the localization precision and make microbubbles more distinguishable from background.

One of the key evolution will probably come from 3D super-resolution ultrasound. The lack of information in the elevation direction is particularly damaging to ultrasound localization microscopy making velocimetry inaccurate, motion-correction incomplete and providing only a single plane per acquisition. Fortunately, new results (Heiles et al. 2018, Heiles et al. 2019, Harput et al 2019) show encouraging 3D reconstruction capabilities (figure 10). However, matrix transducer with numerous parallel acquisition channels are still challenging to fabricate and such technology remain a bottle-neck for 3D ULM, especially at higher frequencies.

Figure 10:

Volumetric Ultrasound Localization Microscopy implemented on an anesthetised rat brain using a 2D matrix array with center frequency 9MHz (Heiles et al. 2018). Voxel size is 10 μm, region of imaging is 13.28mm deep, 10.5mm wide, and 11.4mm in length. The “glow” colormap was used to encode microbubble density. The brighter the color of the voxel, the more microbubbles have passed through that voxel.

Should the field restrict itself to microbubbles? We have seen that nanodroplets or photoacoustic sources could be exploited as localization targets. However, other targets such as cells expressing gas-vesicles could be conceived (Bourdeau et al. 2018, Farhadi et al. 2019), allowing the localization of single cells in the body, maybe opening the field of imaging metastatic cells in-depth.

Should the vasculature remain the only playing field for super-resolution ultrasound imaging? Already, the lymphatic system is being explored (Zhu J. et al., 2019). But the extravascular space could be the next frontier using the natural permeability of tumor vasculature to nano-agents, which could become localization targets after acoustic droplet localization.

Finally, the biggest remaining challenge is probably to define the optimal application of super-resolution ultrasound imaging, the one for which it would become irreplaceable by other imaging modalities. We shall only rest —briefly — when patients’ prognosis will have vastly benefited from the use of ultrasound super-resolution imaging.

Conclusion

In the last 10 years, ultrasound super-resolution has disrupted the diffraction-limit of ultrasound imaging through techniques such as ultrasound localization microscopy. By observing directly the microvasculature in the brain, kidney, skin, lymph nodes, and in tumors in animals and, in some cases in humans, it provided a new window to study the physiology and pathologies of tissue. In-depth microscopy is now achievable with a clinical modality, which can provide a wealth of the new information such as the density and tortuosity of microvessels or even minute modifications in flow patterns. These authors’ collective has participated in the early development of ultrasound super-resolution. Its scope has now widened and international endeavour will define the various clinical applications of the technique, targeting the major killers which affect the smallest blood vessels of the body such as cancer, diabetes (Ghosh et al. 2019), artheriosclerosis. In the future, we foresee new volumetric ultrasound scanner which could map the 3D microvascular of an entire organ, allowing precise diagnosis with reduced user-dependency.