Wavefront shaping: A versatile tool to conquer multiple scattering in multidisciplinary fields

Abstract

Optical techniques offer a wide variety of applications as light-matter interactions provide extremely sensitive mechanisms to probe or treat target media. Most of these implementations rely on the usage of ballistic or quasi-ballistic photons to achieve high spatial resolution. However, the inherent scattering nature of light in biological tissues or tissue-like scattering media constitutes a critical obstacle that has restricted the penetration depth of non-scattered photons and hence limited the implementation of most optical techniques for wider applications. In addition, the components of an optical system are usually designed and manufactured for a fixed function or performance. Recent advances in wavefront shaping have demonstrated that scattering- or component-induced phase distortions can be compensated by optimizing the wavefront of the input light pattern through iteration or by conjugating the transmission matrix of the scattering medium. This offers unprecedented opportunities in many applications to achieve controllable optical delivery or detection at depths or dynamically configurable functionalities by using scattering media to substitute conventional optical components. In this article, the recent progress of wavefront shaping in multidisciplinary fields is reviewed, from optical focusing and imaging with scattering media, functionalized devices, modulation of mode coupling, and nonlinearity in multimode fiber to multimode fiber-based applications. Apart from insights into the underlying principles and recent advances in wavefront shaping implementations, practical limitations and roadmap for future development are discussed in depth. Looking back and looking forward, it is believed that wavefront shaping holds a bright future that will open new avenues for noninvasive or minimally invasive optical interactions and arbitrary control inside deep tissues. The high degree of freedom with multiple scattering will also provide unprecedented opportunities to develop novel optical devices based on a single scattering medium (generic or customized) that can outperform traditional optical components.

Graphical abstract

Public summary

• •Wavefront shaping can be used to achieve optical focusing deep inside scattering media for biomedical imaging, sensing, stimulation, and treatment
• •Wavefront shaping can be used to construct functional devices with scattering media
• •Wavefront shaping can be used to attain linear and nonlinear optical processes through multimode fibers

Introduction

The extremely sensitive nature of optical signals to medium changes or inhomogeneities makes light a promising tool to probe or treat the target medium. In recent decades, the sensitivity of light has been exploited by the invention of many exciting optical techniques and instrumentation, which has contributed significantly to the advancement of modern science and technology. To achieve high spatial resolution, most of these implementations rely on the use of ballistic or quasi-ballistic photons that allow for coherent focusing of light,1, 2, 3, 4, 5, 6, 7, 8 which is not an issue for transparent or thin scattering samples. However, in thick biological tissues or tissue-like scattering media, photons are multiply scattered owing to the inherent inhomogeneous distribution of refractive index. As a result, the number of ballistic photons decreases exponentially with increasing propagation depth; high-resolution optical focusing and imaging in biological tissues are limited to around 1 mm beneath the skin or tissue surface⁹ or can only be applied in weakly scattering samples, whereby adaptive optics can be used to compensate for the aberrations.10, 11, 12 At greater depths of more than one optical transport mean free path, adaptive optics fails as ballistic photons are rare and difficult to screen out from diffusive photons; only fully developed speckles can be observed.

Another scenario in biomedicine where optical speckles dominate is light transmission through multimode optical fibers. Optical fibers have been widely investigated for minimally invasive endoscopic imaging, surgery, and stimulation in biomedicine.13, 14, 15 Among different types of optical fibers, multimode fibers (MMFs) are often preferred because they encapsulate thousands of propagation modes within a small fingerprint (diameter down to tens of micrometers).¹⁶ The applications, however, encounter limitations owing to the mode dispersion and intermodal interference in MMFs, resulting in a speckle pattern that resembles multiple scattering from a thick scattering medium.

Apart from biomedical implementations, multiple light scattering also impacts many other optical applications. For example, transparent or homogeneous optical components, such as lenses, mirrors, polarizers, and beam splitters, are usually desired to ensure minimal optical loss and functionality.¹⁷ To avoid scattering, there are strict requirements on the homogeneity of the refractive index and structure uniformity of the optical materials used in the system; a small irregularity or deformation may lead to considerable degradation in performance. This limits the choice of materials and increases the cost of the device. In addition, the stringent structure requirement to optical components such as waveguides and photonic crystals further constrains the degrees of freedom for light propagation. To summarize, multiple scattering is detrimental to various important optical applications such as the penetration depth of high-resolution optical techniques in biological tissues,³ the applicability of MMFs,¹⁸ and the flexibility in the use of optical components.¹⁹

Given its obvious importance, overcoming multiple scattering has been a long desired goal that unfortunately used to be too challenging, as the process and the consequence of multiple scattering seemed to be random and too complicated to deal with.²⁰ However, in the 1980s, researchers started to notice that the seemingly random scattering events were actually deterministic within the speckle decorrelation window.^21,22 In 2007, Vellekoop and Mosk proposed a game-changing solution called wavefront shaping (WFS), with which the optical scattering-induced distortions can be precompensated by iteratively optimizing the input wavefront23, 24, 25, 26, 27, 28 or time-reversing the scattering wavefront.29, 30, 31, 32, 33 Furthermore, it was shown that the optical propagation through a complex medium can be systematically described by measuring the transmission matrix (TM) of the medium,34, 35, 36, 37, 38, 39, 40 which enables the manipulation of an incident wavefront for any desired output. In the last decade, the field of WFS has demonstrated encouraging developments whereby some methods have already found wide use in research. For example, WFS has been used to compensate for optical scattering to achieve diffraction-limited optical focusing at depths in complex media,41, 42, 43, 44 potentially reshaping the landscape of high-resolution optical applications in biomedicine, such as imaging^18,45 and trapping.^46,47 Wavefront shaping also empowers one to exploit scattering to achieve some unique functions that are otherwise impossible with ballistic light. For example, WFS has been used to reconfigure scattering media into functional components, such as linear operators,⁴⁸ polarization controllers,^49,50 and optical beam splitters,^17,51, 52, 53 which may inspire exciting innovations in optical computation. WFS has also been used to control the mode coupling efficiency and light-matter interaction efficiency to optimize the output light,⁵⁴ such as enhanced or modulated information transmission or communication through heavily scattering media. Last but not the least, WFS also provides promising solutions to overcome the inherent mode dispersion in MMFs,⁵⁵ which may potentially revolutionize the frontiers of fiber endoscopy and communication. To provide insights into how wavefront shaping techniques contribute to related fields, in this review we present the basic concept of multiple scattering and the working principle of WFS, then discuss recent advances of WFS in optical focusing, imaging, and manipulating through/inside scattering media and functionalized devices, as well as light control and modulation in MMFs for versatile applications.

Principle

Optical scattering in disordered media

Light propagation inside structurally disordered media can behave differently from that inside homogeneous media such as air, clean water, and clear glass, and various models have been developed to formulize the strong scattering behaviors.^20,25,56 Propagating for a distance within the transport mean free path (TMFP), light behaves ballistically or quasi-ballistically, i.e., the information transmission between the object and image is one-to-one and therefore localized (Figure 1A). Beyond the TMFP, light propagation becomes random and diffused (Figure 1B) such that one point on an image may find contributions from many points from a source, and vice versa.^7,57 Consequentially, the point spread function of a diffusive system is a random pattern,^35,58 which transforms the ordered information into a seemingly random pattern, i.e., speckles.

Figure 1.

The principle of wavefront shaping

(A) Light propagation in an optically clear system.

(B) Light propagation in a multiple scattering system. A (red) and B (blue) are two point sources illuminated along the optical axis (dashed arrow) on the object plane, whose conjugated points on the image plane are A′ and B′, respectively.

(C) Visualization of the TM model. x_n (j = 1 … N) is the n^th controlled mode in the input field (x), y_m (m = 1 … M) is the m^th output mode in the output field (y), and the relationship between x and y is built by the transmission matrix (T) of the scattering medium with elements t_mn.

Albeit complex, the transformation is deterministic within the speckle correlation time. For a static medium, even if it is highly scattering, the output field in response to the same input field is invariant under identical conjugation constraints. However, once the constraints change as a result of internal motions or external perturbations, the transformation decorrelates rapidly. Before temporal decorrelation occurs, spatially there is a narrow range in which the output field translates/tilts with the input field, i.e., the memory effect (ME).⁵⁹ Within the ME range, imaging against the scattering can be achieved through linear transformation,⁶⁰ but the field of view (FOV) is usually limited (on the order of 10 μm $\times$ 10 μm⁶¹). A thicker medium narrows down the effective ME range and hence the FOV, being a critical obstacle for practical applications. Therefore, estimating the medium response to the input light field is indispensable to revealing information hidden in the scattered output. A rigorous derivation of the scattering models has been provided in Rotter et al.7, 57 and some other earlier works.^20,57,62

WFS via TM engineering and feedback-based optimization

To model the scattering process, speckles can be treated as the interference of numerous independent optical paths or the superposition of their electric fields. In Figure 1C, a complex matrix, i.e., TM, can incorporate the effect of all scattering events between each input-output channel:^35,63 the field at each output channel ($y_m$ element in $Y$, the output field) is a weighted sum of the input channels ($x_n$ elements in $X$, the input field), i.e., $y_m=\sum t_{mn}{x}_n$ or $Y=TX$, where $T$ is the TM with complex elements $t_{mn}$. By manipulating this matrix, information about the disordered medium packed in the matrix can be extracted, and hence either physical or engineering implications can be revealed. Therefore, measuring TM is technically of concern. Two types of approaches, analytical or statistical, have been explored. For analytical solutions, interference is employed: (1) an internal reference pattern is applied on the wavefront modulator together with probing wavefronts via phase-shifting operations;^35,64,65 (2) an external reference beam is introduced to interfere with the output speckle field for holographic recording.^66,67 The other class of measurement is based on statistics, mostly driven by Bayes’ theorem,^68,69 and usually requires more measurements to estimate the TM elements accurately.

With TM measured, the output wavefront can be arbitrarily engineered either globally or locally. Global control is principally based on singular value decomposition. By investigating the eigenvectors of $T^{†}T$ ($†$ denotes conjugate transpose), both the energy delivery (i.e., the so-called eigen-channel)70, 71, 72, 73 and polarization through the scattering medium can be fully controlled.^49,74 Locally, the output light can be confined into a customized pattern, such as single- and multi-point optical focusing,³⁵ via inversion of TM⁷⁵ or directly tuning the phase and/or amplitude of the input field according to the TM elements.⁴¹ Hence, modulated input for desired focusing onto any point on a detection plane can be obtained simultaneously. Raster scanning on the detection plane can be easily realized, and imaging against scattering medium becomes feasible.^16,76,77

Note that instability of the scattering medium or the system could significantly degrade the effectiveness of the measured TM and hence the modulated wavefront.^55,78,79 The input-output mapping is inevitably relaxed for adaptive modulation, and feedback-based optimization is therefore required. These optimization algorithms, including evolutional algorithms,79, 80, 81, 82, 83, 84, 85 artificial intelligence algorithms,86, 87, 88 and their combination,^89,90 can instantly modify the modulation patterns while screening the feedback variations. Encountering stronger variations (e.g., a dynamic medium), optimization with physics prior91, 92, 93 works more efficiently, which quantifies the error in the optimized wavefront for optical focusing, while previously the number of to-be-corrected pixels on a spatial light modulator (SLM) is empirically guessed. Besides, learning-based wavefront optimizations are, in addition, adaptive by decomposing motions into a series of sequential quasi-stationary processes for phase modulation.⁹⁴ These methods, complementing TM-based approaches, have realized promising light focusing and, more importantly, superior noise resistance.

Notably, the application of TM approaches assumes linearity of the investigated system. In contrast, although less capable of channel decomposition or focus scanning, feedback-based algorithms can be more adaptive and versatile, especially when nonlinear effects or excitations govern the detected signals from the scattering system.⁵⁵

WFS via optical phase conjugation

The aforementioned WFS approaches require continuous measurements and updates of feedback signals, and a desired optimization generally needs thousands⁴¹ or even millions⁶⁵ of measurements. Without sophisticated electronic engineering, these methods are deficient in overcoming the rapid changes of optical fields arising from the dynamic medium, such as in vivo biological tissues (decorrelation on the order of milliseconds, decreasing with increasing depth).⁹⁵ A robust solution to this challenge is through optical phase conjugation (OPC), in which direct manipulation of the scattered field rather than introducing precompensation to the incident wavefront is of interest. The first OPC demonstration through a ground glass dates back to the 1960s.⁹⁶ This concept, after nearly half a century of development, was transferred to mitigate the volumetric scattering arising from biological tissue³⁰ and even live animals.⁹⁷

In principle, the scattering process is time reversed: the scattered field is recorded and phase conjugated by a phase conjugate mirror (PCM), and consequently the phase-conjugated field retraces back through the scattering medium, propagating along opposite directions of the incident light and finally converging to the origin of incidence. This phase conjugation implementation allows for reversal of the scattering and hence refocusing of light with merely one snapshot, based on which some successful in vivo demonstrations of fast WFS have been realized.^97,98 The core module, i.e., PCM, is initially implemented analogly with photorefractive materials^30,97 and later digitally with an interference recorder (e.g., a digital camera) pixel-wisely matched with an SLM at the same optical position.^99,100 Either approach yields quick responses (on the order of milliseconds) while the digital OPC, DOPC, suffers no theoretical limit for the maximum reflectivity and works for both continuous-wave and pulsed lasers. Together with correction for minor optical misalignments¹⁰¹ and the intrinsic SLM surface curvature,¹⁰² it provides much better performance. These featured characteristics have made DOPC a popular method to broaden the application to address the scattering in dynamic and optogenetic scenarios.^98,103, 104, 105, 106

Optical focusing and imaging with WFS

Since its introduction to the field in 2007,²⁴ the ability of WFS techniques in the existence of scattering, especially regarding the focusing efficiency and optimization speed, has been greatly enhanced by the accompanying development of various wavefront modulators^98,106, 107, 108, 109 and algorithms.^28,85,93,110 The progress has enabled wide applications, such as optical focusing and imaging inside scattering media, which will be expatiated in this section.

Directly probing or accessing the optical field inside a scattering medium is usually unattainable because light is diffusive. Thus, to achieve WFS at depths in a noninvasive or minimally invasive manner, internal guidestars^9,97,105,111 are indispensable to provide feedback that can gauge the light field in situ within the sample.

In OPC-based wavefront shaping, optical focusing can be achieved with one snapshot by converging conjugated light back to the original internal guidestar. Time-reversed ultrasonically encoded (TRUE) optical focusing was first proposed by using ultrasonic mediation based on the acousto-optical effect as the internal guidestar.^{31,97,112,113} Note that only photons tagged with the ultrasonic frequency shift of f_a can contribute to the hologram recording at the PCM, ensuring that the generated phase-conjugated light converges to the original ultrasound focus, as illustrated in Figure 2A. As ultrasound is scattered ∼1,000 times less than light in biological tissue,³¹ this method can be used for acoustic-resolution optical focusing in deep tissue. Recently, this method was successfully extended for optogenetic modulation of neural activities in 800-μm-thick acute mouse brain slices.¹⁰³ However, the resolution of this method is on the scale of the size of the ultrasound focus, which is considerably larger than the optical wavelengths. To improve the resolution, time reversal of variance-encoded light was proposed, whereby the size of the corrected optical focus can be obtained with speckle-scale lateral resolution (5 μm) by encoding individual spatial modes inside the scattering medium with unique variances.¹¹¹ Apart from ultrasonic mediation, optical perturbation induced by moving absorbers was also used to guide the time reversal in an implementation referred to as time-reversed adapted perturbation (TRAP), as shown in Figure 2B. To control the perturbation movement precisely and remotely in TRAP focusing, magnetic particles controlled externally were proposed to guide optical focusing inside scattering media in a noninvasive manner.^114,115 The diffused light-field perturbation can also be induced in situ by destructing microbubbles with a focused ultrasound beam.¹¹⁶ The size of the optical focus in this approach is ∼2 μm as determined by the size of microbubbles. All of these aforementioned schemes allow for deep-tissue focusing and imaging with high resolution and specificity, which spurs further biomedical applications, such as optogenetics¹⁰³ and photothermal therapy.¹¹⁷

Figure 2.

Optical focusing deep inside scattering media

(A) (a1) Illustration of the TRUE optical focusing concept. (a2) Simulated holograms recorded in a PCM. (a3) Light intensity distribution on plane A by reading the hologram recorded in the PCM.

(B) (b1) Illustration of the concept of TRAP. (b2) Distribution of focal light intensity. (b3) Speckle pattern observed on the SLM surface during the probing process. Scale bars, 500 μm.

(C) (c1) Measurement of a TM via PA feedback signals. (c2) Measured PA trace following a single laser shot through a scattering diffuser. (c3) The measured complex-valued TM gives the influence of each SLM pixel (vertical axis) on each acoustic voxel (horizontal axis).

(D) (d1) Dual-stage optimization of photoacoustically guided wavefront shaping (PAWS). (d2) Optical fields at different stages.

Images in (a1–a3) are from Liu et al.,⁹⁷ (b1–b3) are from Ma et al.,¹⁰⁵ (c1–c3) are from Chaigne et al.,¹¹⁰ and (d1, d2) are from Lai et al.⁴¹

For precompensation-based WFS, such as TM engineering and iterative optimization-based approaches, photoacoustic (PA) signal has proved to be an excellent internal guidestar to gauge localized photon flux inside scattering media. Short laser pulses induce a photothermal effect in the sample and produce so-called PA waves, which can be detected externally by an ultrasound transducer.^6,62,118, 119, 120 PA-guided WFS has been demonstrated in both iterative^41,45,121, 122, 123 and TM-based^110,124,125 (Figure 2C) schemes that allow for noninvasive focusing and imaging at depths. However, such optical focusing ability is limited to the size of the acoustic focal region of the transducer, which contains many speckle grains in the region of the PA guidestar. It has been shown that by using the variations of Gaussian spatial sensitivity of the transducer along the acoustic axis with a genetic algorithm, subacoustic optical focusing can be created.⁴⁵ Moreover, nonlinear PA feedback based on Grueneisen relaxation effect was proposed as the internal guidestar for iterative optimization, which successfully focused light onto a single speckle grain (Figure 2D).⁴¹ Another internal guidestar mechanism exploits the strength of ultrasonically mediated light based on the same acousto-optic effect as that in TRUE.¹²⁶ Such an ultrasonically encoded light mechanism can provide noninvasive feedback that can be used for either iterative wavefront shaping or direct phase conjugation and can achieve optical focusing inside a scattering medium.¹²⁷

WFS is not ultimately constrained to the utilization of guidestars. In recent works, a model-based wavefront shaping has been developed that does not require any physical guidestar,¹²⁸ provided that the refractive index distribution of the sample is available. Assuming a virtual guidestar is placed at any desired position behind the sample, the desired wavefront can be computed numerically and constructed experimentally with an SLM for light refocusing. However, the requirement of prior knowledge about the sample is still difficult to meet in practice, for example biological tissues with dynamically changing refractive index distributions. To overcome such a limitation, an image-guided wavefront shaping approach that allows guidestar-free incoherent imaging has also been reported.¹²⁹ The approach adopts generalized image-based metrics as the iterative feedback to recover a hidden object with the object itself as the guidestar. Such a setting is promising for noninvasive endoscopy.

Functionalized devices with WFS

A scattering medium deranges the spatial/temporal distribution of light, resulting in scrambling of an incident wavefront.^20,24 However, as discussed earlier, this seemingly randomizing effect of disorder on the wavefront is deterministic, and the complex scattering system can be effectively represented by a TM—i.e., a TM can be used to accurately bridge the relationship between the input and output fields. By injecting light into a well-deployed combination of input modes via an SLM, a desired output field can be obtained. In other words, an optical scattering medium can be exploited to perform either specific or multiple functions with the assistance of WFS.

Scattering optical components

Initial efforts of WFS sought to overcome the scrambling of optical wavefronts to form an optical focus behind a scattering medium, whereby the medium can be regarded as a scattering lens in the far field.^24,35,130 This functionality has already been sufficiently discussed in earlier sections. In the near field, a scattering medium can also act as a scattering lens to form a subwavelength focus.^43,131 As shown in Figure 3A, the speckle size is at subwavelength scales^132,133 when near-field scanning optical microscopy (NSOM) is used to record the optical pattern in the near field. In comparison, the speckle grains in the far field are on the optical wavelength scale, as shown in Figure 3A. The near-field TM can then be measured with a conventional NSOM system. The linear relation between the input far field and the scattered output near field allows coherent control of the near field at arbitrary positions, whose full width at half maximum (FWHM) could be as small as 165 nm, as shown in Figure 3A. This is about one-fourth the wavelength of the incident light (633 nm). Apart from usage as a stand-alone scattering lens, a scattering medium can also be used to improve the focusing resolution of a conventional lens.⁴⁴

Figure 3.

Scattering optical components

(A) (a1) Optical speckle measured with conventional optics in the far field. (a2) Optical speckle measured with an NSOM in the near field. (a3) A traditional wavefront shaping setup incorporated with a commercial NSOM system. (a4) SEM image of the NSOM tip aperture. (a5) Far-field image of light radiating from the tip obtained with an objective lens of 0.8 NA. (a6) Measured amplitude and phase of the far-field speckle pattern after light from the NSOM aperture. (a7) Reconstructed image of the same light source using the TM method.

(B) (b1 and b2) Images of an optimized p-polarized focus taken with the analyzer oriented in horizontal (b1) and vertical (b2) directions. (b3 and b4) images of an optimized s-polarized focus taken with the analyzer oriented in horizontal (b3) and vertical (b4) directions. (b5) Polar plot clearly demonstrates the transition from linear to circular polarization and the reversal in linear polarization direction.

(C) (c1) Two incident modes (1 and 2) are phase-modulated with an SLM. (c2) Camera image for two optimized spots when mode 1 is blocked. (c3) Optimized phase pattern on the SLM.

(D) (d1) A transfer lens with a short focal length produces a hologram with a large viewing angle yet small image size. (d2) When a transfer lens with a longer focal length is used the image size becomes larger, but the viewing angle is reduced. (d3) The image size and the viewing angle can be simultaneously increased when a scattering medium is introduced. (d4) By controlling the wavefront impinging on the scattering medium, a specific 3D hologram is generated. (d5) Several letters were projected in a 3D space sequentially.

Images in (a1, a2) are from Park et al.,⁴³ (a3–a7) are from Park et al.,¹³¹ (b1–b5) are from Park et al.,⁵⁰ (c1–c3) are from Huisman et al.,⁵¹ and (d1–d5) are from Yu et al.¹³⁸

In addition to superior lens performance, WFS-assisted scattering medium also holds potential for multidimensional light control, such as for controlling wavelength and polarization degrees of freedom. In traditional optics, one can use a specific optical component to change a certain light property. In scattering optics, the scattering medium can serve as a specific functional component only when the TM contains the corresponding feature of the specific function. For instance, if one wants to use a scattering medium as a polarization-selective component, such as a dynamic active waveplate, the polarization relationship between the input and output fields (i.e., speckles) should be measured.⁵⁰ As shown in Figure 3B, optimized foci with orthogonal polarizations (p and s) at the same position can be switched by displaying corresponding patterns on the SLM after measuring the polarization-included TM. Moreover, shifting the phase of the p-polarized focus while maintaining the phase of the s-polarized focus unchanged results in a shift from linear to circularly polarized foci (Figure 3B). In this scenario, the scattering medium can be used to produce arbitrarily polarized foci with no moving parts. Similarly, when the wavelength-related TM is measured, the scattering medium can function as a spectrum analyzer.134, 135, 136

The WFS-assisted scattering medium can also be used as optical circuits or beam splitters by introducing more independent input channels. Huisman et al. developed programmable optical beam splitters by using a layer of dry white paint in combination with WFS.⁵¹ As shown in Figure 3C, two separate beams were shone through a layer of white paints to create two enhanced spots. Furthermore, the intensity ratio between these two spots can be adjusted by simply adding a phase offset on either beam. This concept has been further extended to generate multiport optical circuits¹⁷ or to control quantum interference.^52,53,137

In another interesting approach, a scattering medium can be used as a dynamic three-dimensional (3D) holographic display component.138, 139, 140 3D holographic display is a disruptive technology for the future display industry that constructs dynamic 3D holographic scenes.¹³⁸ Conventional holographic displays based on refractive optics are limited to small-sized images with a narrow viewing angle owing to insufficient controllable modes on wavefront modulators that are not sufficient to support the needed high space-bandwidth product, which is the product of active aperture and viewing angle.¹⁴¹ This product, also known as étendue, is constant for a single-unit wavefront modulator using a lens to produce a holographic display. If a scattering medium is introduced, the product is no longer constant because the ordered relationship between the input field on the wavefront modulator and the output field on the holographic display is totally scrambled by multiple light scattering. When this scrambled relationship is measured using TM approaches, one can construct the desired volume display with a precalculated pattern displayed on the SLM with significantly extended viewing angle and image size (Figure 3D). Yu et al. showed that the product of viewing angle and image size can be enhanced by a factor of 2,600 by adding two holographic diffusers compared with the case of using a single wavefront modulator only.¹³⁸ For example, as shown in Figure 3D, two characters “3D,” with 15 foci for each character, are displayed on different output planes. These characters cover a very large display volume while maintaining a high resolution of each generated focus of only 1 μm.

Reconfigurable optical computing units

It should be noted that only several output channels are utilized in the aforementioned scattering optical components, which only take advantage of the corresponding several rows of the TM. However, for a scattering medium the TM is of large dimension. Moreover, the lack of symmetry of the system further increases the rank of the TM. Hence if more output channels are adopted, the scattering medium can function as much more complex components, such as a linear operator as proposed by Matthès et al. by searching optimum input-output projectors (Figure 4).⁴⁸ The procedure is briefly explained as follows. First, the scattering medium’s complex-valued TM is measured without interferometric measurements by using a phase retrieval algorithm (Figure 4A).⁶⁹ A mixed-integer convex solver algorithm¹⁴² is then used to find an approximate input mode to minimize the difference between the target operator and the measured operator (Figure 4B). Lastly, once the appropriate input mode is calculated and displayed on the dot matrix display (DMD) to conduct optical analog computation (Figure 4C), the scattering medium can act like any desired linear operator, for example a Fourier transform operator as shown in Figure 4D. Most recently, Yu et al. demonstrated that a single scattering medium assisted by wavefront shaping can also perform basic optical logic operations.¹⁴³ These studies have suggested the great potential of developing reconfigurable optical computing units with a scattering medium empowered by WFS.

Figure 4.

Reconfigurable optical linear operations

(A) System calibration.

(B) Optimal input projection calculation.

(C) Analog computation.

(D) Experimental results of a 16 × 16 discrete Fourier transform.

All images are from Matthès et al.⁴⁸

Modulation of multimode fibers via WFS

In this section, we first review the optical modulation inside MMF with WFS involving in the linear regime and nonlinear regime, respectively. The classification basis of the linear and nonlinear ones is whether the process is involved in nonlinear optical process. To be specific, in the nonlinear regimes some nonlinear effects inside MMF, such as stimulated Raman scattering and four-wave mixing, can be modulated. Moreover, owing to the flexibility and small-diameter features of MMF, a series of notable unique implementations have been enabled based on the integration of MMF and wavefront shaping techniques. In the last part of this section, two representative application scenarios (optical endoscopic imaging and optical tweezers) are discussed.

Linearity modulation

When nonlinear processes are not generated within the fiber, the input and output of an MMF can be linearly bridged with the TM model, with which focusing and scanning of the focus within the measured FOV can be achieved, as illustrated in Figure 5A. A lensless MMF endoscope is, therefore, enabled,¹⁴⁴ based on which in vivo imaging of neurons and blood cells in the mouse brain have been reported.^76,77 The ability to transform any spatial mode as a focus at a specific position also turns the SLM-integrated MMF into a mode sorter, as shown in Figure 5B. Light coupled into specific spatial modes, such as Fourier, Laguerre-Gaussian, random basis, and optical angular momentum, can be sorted by an MMF when optimized phase masks are applied.^145,146

Figure 5.

Linearity modulation in MMF

(A) Focusing and focus scanning at the distal end of an MMF. (a1) Foci at the distal end of MMF indicating the spatial power uniformity. (a2) Zoom-in intensity distribution and azimuthally averaged profile of a focus within the blue square in (a1).

(B) Mode sorting. (b1) Fourier basis added into the optimized phase mask. (b2) Modes can be spatially sorted by focusing at different positions (b3).

(C) Polarization control. (c1) Fiber depolarization. (c2) Polarization control by WFS. (c3) Experimental demonstrations of overcoming fiber depolarization and complete conversion to orthogonal polarization.

Images (a1, a2) are from Turtaev et al.,⁷⁷ (b1–b3) from Defienne and Faccio,¹⁴⁵ and (c1–c3) from Xiong et al.⁴⁹

Polarization scrambling in MMF is another concern because of the inevitable imperfection of fabrication and external perturbations, but can be effectively addressed by controlling the phase of the input beam. For example, the mode coupling in MMF surprisingly serves as a bonus to address the depolarization, since it provides sufficient degrees of freedom to control the polarization for every output. As shown in Figure 5C, the polarization of the output channels can be actively tuned by shaping the input with a polarization-resolved TM, which is independent of the input polarization.⁴⁹ Among these realizations, the multiple modes in MMFs determine the feasibility, although this has previously been considered to be a defect compared with single-mode fibers (SMFs). Indeed, the number of modes in MMFs defines the degree of freedom for the WFS.⁷⁷ For specific target applications, the number of modes can be customized by simply defining the structure of the MMF. In this regard, WFS matches the features of MMF, based on which many more interesting implementations have been inspired, as discussed in the following.

Nonlinearity modulation

Besides the linear regime, spatiotemporal nonlinearities related to MMFs, graded-index multimode fibers (GRIN-MMFs) in particular, have also drawn extensive attention for both fundamental research and applications. In addition to the high energy capacity, GRIN-MMFs have two other unique features of interest: (1) reduced modal dispersion: interaction among fiber modes is retained in a relatively long propagation distance, which is available even for the high-energy ultrafast regime; (2) spatial self-imaging: the beam width and intensity periodically oscillate along the fiber and the refractive index can be periodically modulated along the longitudinal axis by the Kerr effect. With a high-power ultrafast pulse beam, nonlinear phenomena arise in GRIN-MMFs, such as graded-index solitons, spatiotemporal instability, self-beam cleaning, and supercontinuum generation. Manual adjustment of the coupling of lens and fiber makes the control over these nonlinearities possible.¹⁴⁷ To flexibly control and customize these nonlinearities, WFS can be utilized by shaping the input beam via an SLM.

In the linear regime, TM plays a very important role in mapping the input-output relation, but the applicability of the TM decreases significantly when nonlinear effects arise. Therefore, a modulation mechanism that does not assume linearity is desired. In this regard, deep-learning and optimization algorithms have been utilized to harness nonlinearities. With a deep neural network (DNN) whose input is the target spectrum and the output is the intensity profile of the fiber input, frequency conversion within an MMF can be achieved.¹⁴⁸ As shown in Figure 6A, the intensity patterns of the input light to be displayed on the SLM (insets) can be predicted by a well-trained network, while the experimentally generated Raman scattering and supercontinuum (solid red lines) from the MMF match quite well with the target spectrum (dashed lines). Adaptive algorithms, such as genetic algorithm (GA), can also nudge the nonlinear process to realize versatile spectrum manipulations.⁵⁵ For example, by evolving the phase pattern on the SLM, the modulated light can enhance the four-wave mixing peak at 518 nm by 6-fold (Figure 6B) and shift the fundamental mode (e.g., LP₀₁ in Figure 6C) at arbitrary wavelength induced by the stimulated Raman scattering. Furthermore, this study also enables flexible spectrum manipulations such as shifting and suppressing anti-Stokes peaks (Figure 6D). In an even more complicated scenario such as a multidimensional fiber laser, nonlinearities can be tackled by optimizing the wavefront with proper cost functions.⁵⁴ For example, when working in a quasi-continuous-wave mode, the speckle output from a fiber laser can be shaped into a Gaussian profile (Figure 6E). WFS also enables mode locking to tune pulse laser excitation, pulse width, and generation of multiple pulses (Figure 6F). These studies validate the feasibility of WFS-assisted MMF for nonlinear control. It should be noted that the nonlinear excitation from the pump signals involves complicated physical mechanisms. Nevertheless, the physical information can be indicated in the refined wavefront or mode of the pump beams during the learning or optimization process. Such convenience thereby extends the application of the wavefront shaping across both linear and nonlinear regimes.

Figure 6.

Nonlinearity modulation in MMF

(A) Learning-based spectrum control. (a1) Stimulated Raman scattering (SRS) generation under 85 kW peak power. (a2) Supercontinuum generation under 150 kW peak power. Dashed and solid lines are the designed and measured spectra, respectively, and the insets are the designed pattern, as predicted by the DNN. Scale bars, 3 mm.

(B) WFS of four-wave mixing. (b1) Photon counts of FWHM peak at 518 nm with and without optimization. (b2) Fiber outputs at Stokes, pump, and anti-Stokes modes.

(C) Wavefront shaping optimization of SRS at the third (c1) peak in the fundamental LP₀₁ (right inset), and the near-field images of lower-efficiency, 543-nm SRS peak corresponding to different Raman cleaning processes (c2).

(D) WFS of spectral shifts. (d1) With optimization, the spectrum is shifted by up to 20 nm and the gray region is the selected spectral region of interest (ROI) to be optimized. (d2) Energy distribution with and without optimization for suppressing higher-order SRS cascade.

(E) Controlling a multidimensional fiber laser: mode profiles before (e1) and after (e2) GA optimization, as well as the 1D profiles for the highlighted regions (e3).

(F) Mode locking manipulation: temporal profile (f1) and optical spectra (f2) before and after mode locking, pulse width characterized by the autocorrelation of the chirped and dechirped pulses (f3), and generation of multiple pulses (f4).

Images in (A) are from Teğin et al.,¹⁴⁸ (B), (C), and (D) from Tzang et al.,⁵⁵ and (E) and (F) from Wei et al.⁵⁴

Multimode fiber-based applications

Optical endoscopic imaging

Endoscopes are widely applied in medical diagnosis and treatment of internal tissue structures. Realizations based on WFS-empowered MMFs enable extreme miniaturization of the probe (diameter <100 μm) and relax the pixelation problem encountered in fiber-bundle-based demonstrations.¹⁴ As discussed earlier, calibration of the TM of MMFs allows for arbitrary light focusing and raster scanning at the distal end of the fiber. If the scanning foci are targeted on fluorescent beads placed behind the fiber, the excited fluorescence signal from each focus can be collected by the same fiber and processed to form an image.⁴⁷ Such capability can also be extended to in vivo imaging, since the compact size of MMFs is ideal for minimally invasive intervention into deep tissue. For example, Piestun’s group demonstrated a single MMF endoscope¹⁶ and a novel design of endomicroscope for imaging neural activity of biological samples.⁷⁶ Moreover, in 2018 Čižmár’s group reported the first in vivo deep-brain imaging using MMF endoscopy (Figure 7A), which achieved subcellular resolution as shown in Figure 7B.⁷⁷

Figure 7.

MMF-based endoscopies

(A) The configuration for deep-brain imaging using an MMF.

(B) In vivo images of neuronal somata and processes, as well as hemorrhage of cortex captured during the insertion of an MMF probe.

(C) Illustration of MMF-based far-field endoscope.

(D) Endoscopic imaging of a mechanical clock and a sweet pepper.

(E) Experimental setup for hybrid fluorescence-photoacoustic imaging with an MMF.

(F) False-color hybrid imaging of a red blood cell (in red) and fluorescence particles. Scale bar, 30 μm.

Images in (A) and (B) are from Turtaev et al.,⁷⁷ (C) and (D) from Leite et al.,¹⁴⁹ and (E) and (F) from Caravaca-Aguirre et al.¹⁵⁶

Efforts have been made not only to improve the spatial resolution and frame rate of MMF-based endoscopes to support in vivo experiments but also to extend the working distance and FOV to make the technique more suitable for practical applications. For example, most recently a far-field endoscope using two MMFs, with one for illumination and the other one for the signal collection, has been reported to realize reflectance imaging of distant macroscopic objects (Figure 7C).¹⁴⁹ The FOV of the endoscope scales linearly with the working distance, which could be tens to hundreds of millimeters. Far-field images of 3D natural scenes such as sweet pepper and a mechanical clock were demonstrated (Figure 7D), which, if further engineered, is promising for organ- or tissue-scale imagery. Furthermore, with time-of-flight information, depth perception in macroscopic 3D scenes can be achieved.¹⁵⁰

While promising, the aforementioned MMF-based endoscopies all require precalibration of the fiber and work under the assumption of a rigid endoscope that assumes no obvious deformations such as bending or twisting applied to the fiber. To address this, recent efforts have been made to design flexible MMF endoscopes in which the employment of GRIN fibers yields less influence by bending deformations.^16,151 Another possible solution is to calibrate an MMF in situ with measurements at the proximal end only.152, 153, 154 Notably, however, these setups are technically challenging. More recently, Li et al.¹⁵⁵ simplified the technical dilemma by utilizing the memory effects in structures of arbitrary geometry to efficiently estimate the TM, albeit with limited accuracy.

MMF-based endoscopes are also capable of combining fluorescence and photoacoustic imaging156, 157, 158 for dual-modality imaging. As shown in Figure 7E, a hybrid photoacoustic-fluorescence endomicroscope consists of an MMF and an SMF for fluorescence and PA signal detection, respectively. The imaged red blood cells and fluorescence particles, as shown in Figure 7F, demonstrate the feasibility of the ultrathin hybrid probe. The capability of imaging through MMFs has also been extended to visual confocal filtering,¹⁵⁹ two-photon imaging,¹⁶⁰ multiphoton imaging,¹⁶¹ light-sheet microscopy,¹⁶² and second-harmonic generation imaging.¹⁶³ These integrations have promoted advancements in resolution, contrast, and FOV, which further enable more exciting applications such as stimulation and monitoring of nerve cells.164, 165, 166

Optical tweezers

Intervening in biological processes and systems via optical forces in vivo avoids physical contact and damage to live organisms. In this regard, holographic optical tweezers (HOTs)¹⁶⁷ can also be realized in biological tissue, where the highly focused laser light generated by wavefront shaping allows for precise manipulation of small objects at depths. The compact size of MMFs further extends the working depth of HOTs with minimal invasiveness. For example, microbeads can be manipulated individually after measuring the complex light propagator of the fiber,⁴⁷ or multiple (up to 16) microbeads on the focal plane can be controlled simultaneously by changing the multifocus intensity patterns.¹⁶⁸ Note that the low numerical aperture (NA) of MMFs limits the axial trapping ability, as the axial gradient force cannot balance the radiation force. To resolve this limitation, Leite et al.⁴⁶ designed a soft-glass step-index MMF to enlarge the NA up to 0.96. The mode-dependent loss (MDL) in such a high NA MMF becomes more apparent and yet can be precompensated by the SLM to equalize the power spectrum in the distal far field of the MMF, as shown in Figures 8A and 8B. The obtained high-quality beam after MDL compensation thus allows for multiple optical traps in the lateral and axial directions to confine particles to form 3D cubes (Figures 8C and 8D).

Figure 8.

MMF-based optical trapping

(A and B) The compensation for the MDL of a high NA MMF allows equalizing of the power spectrum in the distal far field of MMF.

(C and D) Multiple HOTs to confine microbeads in square grids and rotating 3D cubes.

Images are from Leite et al.⁴⁶

The previous examples represent configurations designed for highly scattering media. To implement similar ideas inside living tissue, dynamic aberrations from the biological environment and deficiency of ideal internal guidestar are remaining major obstacles. Going further, a single MMF, if the TM is calibrated and updated sufficiently fast, can serve multiple functions simultaneously, such as imaging with trapping, imaging with stimulation, and imaging with treatment.

Discussions and perspectives

Multiple scattering of light has long been regarded as a fundamental barrier that plagues the performance of optical systems. This article reviews the emergence and active ongoing developments in WFS that have opened up many promising avenues where multiple scattering can be tamed or even exploited to fully recover or enhance the performance of optical systems. A few more points, however, should be discussed or clarified before concluding.

As mentioned in the introduction, WFS shares roots with adaptive optics, which has a more extended history and a similar goal: enhancing the transmission and/or receiving of light through distortions. The spatial inhomogeneity of refractive index is the common source of distortions, yet the variance in the severity of distortions results in fundamental differences in what can be achieved in the respective scenarios. For example, in the realm of adaptive optics, distortions in the optical phase can be effectively described using Zernike polynomials as a basis set.¹⁶⁹ Low-order Zernike polynomials, typically up to the 20^th–30^th order, can effectively describe the wavefront distortions.

The situation is quite different in WFS where the amount of turbidity or wavefront distortions has been significantly increased, resulting in multiple scattering of light. The sheer number of modes needed to describe the input and output relations through a scattering medium increases as the number of independent incident modes (channels) is given by N = 2πA/λ², where A is the surface area of the medium and λ is the wavelength of light.¹⁷⁰ Considering that imaging is essentially a low-pass filtering process in which the information contained in the high spatial frequencies is lost and the number of pixels in the sensor is limited, it is technically impossible to measure all modes of the scattering medium.^153,154 For example, although WFS correction of up to several million modes has been demonstrated,⁶⁵ it is still far from perfectly controlled. This can be interpreted as obtaining only a negligible Strehl ratio for existing WFS implementations whereby the background speckles constitute most of the energy. However, not all is lost in multiple scattering. While the Strehl ratio does not hold much meaning in this context, the intensity at the desired focal region can be enhanced by up to 100,000 times.⁶⁵ Such capability provides sufficient contrast for imaging, sensing, stimulation, treatment, and information delivery, which may potentially revolutionize the utilization of light for high-resolution applications at depths.

Another essential and paradoxical aspect unique to wavefront shaping is that the large number of orthogonal modes generated in multiple scattering allows the realization of new applications that were not possible using conventional optical components. Given that conventional optical components based on optical refraction or diffraction are typically designed for a specific purpose, their functionalities are usually fixed. With wavefront shaping, multiple scattering of light connects the incident light to many different output modes that can be utilized for various functionalities.¹⁷¹ We can measure the TM of a scattering medium for various degrees of freedom of light such as polarization, wavelength, spatial frequency, or angular momentum. In other words multiple scattering, which at first sight looks like a hopeless scrambling of information into meaningless speckles, can actually be used to sense or control the many degrees of freedom of light to realize different functions, such as waveplates,^50,172,173 spectrometers,^134,174,175 near-field lenses,^43,176 holographic displays and sensors,^138,139,177 and orbital angular momentum demultiplexers.¹⁷⁸ This is in stark contrast to conventional optical components that need to be carefully designed and manufactured to realize the respective purposes. Moreover, since the calibration and usage of such systems can be realized under well-defined or preferable environments, existing challenges associated with medium stability and/or calibration time consumption can become a matter of choice that can be balanced between application, complexity, and cost. For example, most recently, metasurface¹⁷⁹ and liquid crystal geometric phase diffusers¹⁸⁰ have been introduced to the field to tackle the calibration time consumption issue. The breakthrough of the applications of wavefront shaping, at least for static and moderately dynamic scenarios, is on the horizon.

For bioimaging, however, the situation needs to be further improved to harness the full power of WFS. Here, as the goal is noninvasive or minimally invasive in vivo imaging at depth in tissue, the system must be optimized to best fit specific conditions of the sample which cannot be compromised, such as the following critical challenges. (1) Live tissue is dynamic, which randomizes the TM of the medium with time. The measurement of TM or iterative optimization therefore must be completed within the decorrelation time of living tissue, which is in the millisecond range for many types of organs.¹⁸¹ (2) For fast measurements, single-shot approaches such as employing Shack-Hartmann wavefront sensors are preferred. Note that, however, for measurements in deep biological tissue, Shack-Hartmann sensors are not suitable because their operation principle does not work when only speckles are available.¹⁸² (3) Fast measurements of signals originating from deep-tissue regions result in a low signal-to-noise ratio, which may be challenging for the calibration or optimization process. (4) The inhomogeneous spatial distribution of cells and subcellular organelles constituting biological tissue requires different wavefront corrections for different FOVs,183, 184, 185, 186, 187 bringing further challenges to the viability of fast imaging within the speckle decorrelation time.

Innovations in WFS have naturally been focused on addressing these challenges. To enable faster measurements and more efficient wavefront control, researchers in the community have readily adopted technical advancements in an ongoing effort. For example, the type of wavefront modulators used in WFS has evolved from liquid crystal on silicon SLMs²⁴ to microelectromechanical system mirrors,¹⁸⁸ DMDs,¹⁸⁹ and micromechanical grating light valve technology in a fierce battle to catch up with the decorrelation time of living biological tissue. To increase the resolution and signal-to-noise ratio at depths in vivo, WFS has been extended to longer optical wavelength with multiphoton microscopy.^188,190, 191, 192 It must be clarified that thus far, many of these new WFS implementations have only been demonstrated in the proof-of-principle phase. Nevertheless, continuous progress in merging, modifying, and advancing these efforts toward robust performance in real in vivo experiments will likely boost the second wave of developments in WFS, which may change the landscape of biomedical optical research and practice at the tissue level.

Conclusions

In the past 15 years (2007–2022), we have witnessed rapid advances in the field of WFS in virtually all technological aspects. The initial aim of generating a bright focus through scattering media²⁴ was soon followed by a demonstration of image delivery through turbidity based on the TM formalism.³⁵ Synergy with internal guidestars, such as ultrasonically labeled light and photoacoustic emission, allows for the generation of reference beacons arbitrarily located in deep tissue through holographic methods.^193,194 Development of faster schemes and efforts to increase the effective FOV are actively ongoing.^128,129,195, 196, 197

Thus far, WFS is the most likely solution, albeit not yet perfect, for noninvasive or minimally optical-resolution applications at depths in tissue, where conventional approaches encounter limitations owing to the lack of sufficient ballistic or quasi-ballistic photons. Under such extreme conditions, accurate wavefront measurement and control is desirable yet highly challenging.¹⁹⁸ Despite the remaining difficulties, the rapid growth in different aspects of wavefront shaping is encouraging. The progress is in part due to the adaptation and further development based on the foundations in related fields, such as adaptive optics in astronomy,¹⁹⁹ random matrix theory,^200,201 and time reversal in acoustics^202,203 and photoacoustics.^204,205 Moreover, advances in optimization algorithms and recent applications of deep learning are expected to play a crucial role in realizing faster and more efficient WFS.^88,206,207 Looking back and looking forward, we believe that WFS holds a bright future that will open up new avenues for noninvasive or minimally invasive optical interactions and arbitrary control inside deep tissues. The high degree of freedom in multiple scattering and WFS will also provide unprecedented opportunities to develop novel optical devices based on a single scattering medium (generic or customized) that can outperform traditional optical components. At present, we are at a critical stage in advancing the proof-of-principle innovations in the field toward real-world applications, and we envision many exciting developments yet to come.

Published Online: August 2, 2022

Lead contact website

https://www.polyu.edu.hk/bme/people/academic-staff/dr-puxiang-lai/.