High-resolution limited-angle phase tomography of dense layered objects using deep neural networks

Significance

We demonstrate that it is possible to use deep neural networks to produce tomographic reconstructions of dense layered objects with small illumination angle as low as 10 °. It is also shown that a DNN trained on synthetic data can generalize well to and produce reconstructions from experimental measurements. This work has application in the field of X-ray tomography for the inspection of integrated circuits and other materials studies.

Abstract

We present a machine learning-based method for tomographic reconstruction of dense layered objects, with range of projection angles limited to $\pm {\mathrm{10}}^{○}$. Whereas previous approaches to phase tomography generally require 2 steps, first to retrieve phase projections from intensity projections and then to perform tomographic reconstruction on the retrieved phase projections, in our work a physics-informed preprocessor followed by a deep neural network (DNN) conduct the 3-dimensional reconstruction directly from the intensity projections. We demonstrate this single-step method experimentally in the visible optical domain on a scaled-up integrated circuit phantom. We show that even under conditions of highly attenuated photon fluxes a DNN trained only on synthetic data can be used to successfully reconstruct physical samples disjoint from the synthetic training set. Thus, the need for producing a large number of physical examples for training is ameliorated. The method is generally applicable to tomography with electromagnetic or other types of radiation at all bands.

Tomography is the quintessential inverse problem. Since the interior of a 3-dimensional (3D) object is not accessible noninvasively, the original insight of tomographic approaches was to illuminate through from multiple angles of incidence and then process the resulting projections to reconstruct the interior slice by slice (1–3). In the simplest case, when diffraction is negligible and the illumination is collimated, as is generally permissible to assume for X-ray attenuation (4–7) and electron scattering (8–10) in the far field and for features of size $\sim$1 $\mathrm{\mu }$m and above, the object’s interior is represented by its Radon transform (11) of line integrals along straight parallel paths. The interior of the volume is then reconstructed by use of the Fourier-slice theorem for the Radon projections. On the other hand, if the X-ray beam is not collimated but spherical, then the slice-by-slice approach is no longer applicable and full volumetric reconstruction is required (12, 13). Even when the object is available for observation from the full $360^{○}$ range of projection angles, these instances of tomography are all highly ill-posed because the Fourier-slice property results in uneven coverage of the Fourier space with the high spatial frequencies ending up underrepresented. Ill-posedness increases when the angular range is limited because then an entire cone of spatial frequencies goes missing from the measurement. Alternatively, in this case, tomosynthesis (14) utilizes sheared (rather than rotated) projections to bring slices from within the interior into focus, but with lower contrast since emission from the rest of the volume remains as background.

Additional challenges occur when the inverse problem of interest is to reconstruct in 3D the index of refraction, rather than the attenuation. If the object features are large enough compared to the wavelength, such that diffraction may still be neglected, and the index variations through the object volume are relatively small, then each projection may be modeled as a set of Fermat integrals of phase delay along approximately straight lines. The phase integrals may be obtained, for example, using holographic interferometry (15, 16) or transport of intensity (17). For smaller-sized features and still assuming weak scattering (first-order Born approximation), the projection integrals are instead obtained along curved paths on the surface of the Ewald sphere, a method referred to as diffraction tomography (18, 19). By decoupling the problem into 2 parts, first phase projection retrieval, followed by tomography, these approaches enjoy the benefit of using the advanced algorithms in the 2 respective research fields. However, there is also the danger that errors generated independently during each step may amplify each other. Finally, when strong scattering may no longer be neglected, all 2-step approaches become questionable because the interpretation of the first step as line integrals is no longer valid.

Generally, ill-posed inverse problems are solved by regularized optimization. If $f$ is the object and $g$ the measurement, then the object estimate $\widehat{f}$ is obtained as (20, 21)

$$\widehat{f}=\underset{f}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\left\{{\Vert Hf-g\Vert }^2+\alpha \mathrm{\Phi }\left(f\right)\right\}.$$ [1]

Here, $H$ is the forward operator relating the measurement to the object, $\mathrm{\Phi }$ is the regularizer expressing prior knowledge about the object, and $\alpha$ is the regularization parameter controlling the competition between the 2 terms. The prior may be thought of as rejecting solutions to the inverse problem that are known to violate known properties of the class of objects being imaged; for example, if the class where $f$ belongs is known to have sharp edges, then the regularizer should be applying a high penalty to blurry solutions $\widehat{f}$. Thus, the inherent uncertainty due to ill-posedness is reduced. Sparsity-promoting compressive priors (22–25) found some of their first successes in tomographic reconstruction (26, 27). Compressive sensing is directly implemented through a proximal gradient solution to Eq. 1 if a set of basis functions where the object class is sparse is a priori known. Alternatively, if a database of representative objects is available, then these examples may be used to learn the optimal set of basis functions as a dictionary (28, 29).

Rapid recent developments in the field of machine learning, and deep neural networks (30) (DNNs) in particular, have provided an additional set of tools and insights for inverse problems. It may be shown (31, 32) that recurrent or unfolded multistage DNN architectures are formally equivalent to the iterative solution to the inverse problem in Eq. 1 where the prior $\mathrm{\Phi }$ need no longer be known or depend on sparsity; instead, examples guide the discovery of the prior through the DNN training process. Simpler learning architectures, where $g$ is fed to the DNN directly or after first passing through a preprocessor, have been used for retrieval of phase from intensity (33–36); 3D holographic reconstruction (37–39); superresolution photography (40–42) and microscopy (43); imaging through scatter (44–47); and imaging under extremely low light conditions in the 3 contexts computational ghost imaging (48), consumer-camera photography (49), and phase retrieval (50).

Multistage DNN architectures have been shown to yield high-quality reconstructions in numerous Radon tomography configurations (32, 51–56). Recently, Nguyen et al. (57) used the inverse Radon transform for optical tomography with a single-stage DNN intended to partially correct for the assumption of line integrals breaking down.

In this paper, we apply a Fourier-based beam propagation method (BPM) (58) as a preprocessing step immune from any Radon assumptions. The strongly scattering object is illuminated by a parallel beam under a limited angular range of $20^{○}$, i.e., $\pm 10^{○}$ from the reference axis. Unlike the earlier works on refractive index tomography referenced above, we do not perform a phase retrieval step; rather, the intensity measurements are preprocessed to produce directly an initial crude 3D guess of the object’s interior. This crude guess is then fed to our machine-learning algorithm. The preprocessing step is necessary because, even if we did convert intensity to phase, the results would not be interpretable as line integrals under our experimental conditions. Moreover, by merging phase retrieval and tomography into a single step, our algorithm becomes less sensitive to error accrual.

Large datasets, typically consisting of more than 5,000 examples, are generally required for DNN training. That is feasible in many cases through spatial light modulators (33, 57). However, in many cases of interest spatial light modulators have insufficient space-bandwidth product or are unavailable (e.g., in X-rays); and alternatives to generate physical specimens are expensive or restricted due to proprietary processes. Instead, our approach is to train the DNN on purely synthetic data with the rigorous BPM forward model and then use a physical test specimen (phantom) to test the reconstruction quality with well-calibrated ground truth in experiments.

We chose to design our phantom as emulating the 3D geometry of integrated circuits (ICs). These would normally be inspected with X-rays, so we scaled up the feature dimensions in the phantom for visible wavelengths. The advantage of this choice is that IC layouts provide strong geometrical priors, e.g., Manhattan geometries, and our phantom also exhibited large spatial gradients and refractive index contrast to strengthen scattering. Thus, our methodology is directly applicable to all cases of tomography at optical wavelengths, e.g., 3D-printed specimen characterization and identification and biological studies in cells and tissue with moderate scattering properties. In each case, testing the ground truth would require the fabrication or the accurate simulation of different phantoms meeting the corresponding priors.

There is also value in the study of emulating X-ray inspection of ICs at visible wavelengths, as extensive outsourcing by the IC industry has created a growing concern that the ICs delivered to the customer may not match the expected design and that malicious features may have been added (59). However, in our emulation the phase contrast of the features against the background and the Fresnel number are both higher than typical corresponding IC configurations even at soft X-ray wavelengths.

One advantage of deep learning for inverse problems is speed. Solving [1] separately for each instance of $g$ is computationally intensive, and training a DNN is even more so. This is because both operations are iterative, and the latter is run on large datasets. On the other hand, once the DNN has learned the inverse map from the preprocessed version of $g$ to $\widehat{f}$, the computations are feedforward only. For example, the IC layout priors we exploit here could, in principle, also be learned by dictionaries—but, under strong scattering conditions, the latter would require iterative optimization of Eq. 1 with the forward operator $H$ itself consisting of an expensive computational procedure in each iteration. In our approach, the preprocessing performed prior to the DNN is the most time-consuming operation, and therefore we aim at simplifying the preprocessing step as much as we can, i.e., tolerate a crude approximation, and leave it to the DNN to correct it. In our case, the execution time is 51 s (out of which only 300 ms are taken by the DNN, with the rest being the preprocessor), while learning tomography (60, 61), which is based on a similar gradient descent algorithm, takes 212 s and yields inferior reconstructions (see Fig. 4 for this, and see Materials and Methods for the computing hardware implementation).

Fig. 4.

(A–D) Proximal gradient descent with TV regularization, $K=30$ iterations, for each layer 1 to 4. (E–H) Approximants generated from the experimental measurements with $K=1$. (I–L) Approximants generated from the experimental measurements with $K=8$. (M–P) Reconstructions from the DNN of each approximant E–H, respectively. (Q–T) Reconstructions from the DNN of each approximant I–L, respectively. (U–X) Idealized ground truth obtained from the sample specifications for layers 1 to 4. Note that the color bar range covers more than the range of the data, so there is no saturation effect on the images.

Optical Experiment

We prepared a series of 4 glass wafers with etched structures representing patterned layers from an actual IC design. A schematic cross-section of the sample is shown in Fig. 1A. The glass plates are held together and aligned in a custom-made holder. Immersion oil is added between the plates to minimize parasitic reflections and also tune the phase shift associated with the patterns. The pattern depth was measured to be 575 nm, yielding a phase shift of −0.33 rad for the particular oil used. Note that the phase shift is negative as the refractive index of the oil is lower than the refractive index of the glass. Details about the sample preparation and phase-shift measurements are given in Materials and Methods. The particular patterns etched on the sample are shown in Fig. 1B.

Fig. 1.

(A) Sample cross-section. The depth of the etched patterns was measured (Materials and Methods) and the refractive index of the oil was controlled to achieve a known phase shift of −0.32 rad. $\mathrm{\Delta }z=0.5$ mm. (B) IC patterns used for each of the 4 layers. The white background represents the original wafer thickness and the black areas indicate where the wafer has been etched.

The experimental apparatus is detailed in Fig. 2. A collimated monochromatic plane wave from a continuous wave (CW) laser is incident on the sample, which is mounted on a 2-axis rotation stage. The sample is imaged through a demagnifying telescope to increase the field of view. The detector (an EM-CCD camera) is defocused from the image plane to simulate free-space propagation in an X-ray experiment where no imaging system can be used. Further details are given in Materials and Methods.

Fig. 2.

Experimental apparatus: spatial filter and beam expander. L1 is 10$\times$, 0.25 numerical aperture objective; L2 is a $100$-$\mathrm{m}\mathrm{m}$ lens, with a $5$-μm pinhole F1 in the focal plane; L3 is a 200-mm lens; and L4 is a 100-mm lens. Aperture A1 cuts the outer diffraction lobes of the beam. The sample is mounted on a 2-axis rotation stage rotating along the $x$ and $y$ axes. The sample middle plane is imaged using a telescope lens system with magnification $0.50\times$. The camera is defocused by a distance $\mathrm{\Delta }z=58\hspace{0.17em}\mathrm{m}\mathrm{m}$ from the image plane.

The strength of diffraction effects can be quantified with the Fresnel number $F=a^2/\left(\lambda d\right)$, where $\lambda$ is the wavelength, $d$ the propagation distance, and $a$ the characteristic feature size of the object. The smaller the Fresnel number is, the stronger the effects of diffraction. For the glass phantom considered here and a defocus of 58 mm, $F=0.7$ for the smallest features and $F=5.5$ for the largest. The diffraction pattern is digitized on the camera for different sample orientations. This series of measurements is then passed through a numerical algorithm, described in the next section, whose aim is to yield a first approximate reconstruction, hereafter referred to as the “approximant.”

Computation of the Approximant

As mentioned above, the task of the DNN is significantly facilitated if the raw measurements are preprocessed to give an approximation of the solution. We use a simple gradient descent method to generate the approximant of the sample refractive index distribution.

Light propagation through the object can be computed using the following split-step Fourier BPM. In this method each sample layer $l$ is modeled as a 2D complex mask $f_l\left(x,y\right)=\exp \left[{\alpha }_l\left(x,y\right)+j{\phi }_l\left(x,y\right)\right]$ and the space between 2 successive layers as Fresnel propagation through a homogeneous medium whose index of refraction equals the average refractive index of the sample, as

$$u_l={\mathcal{F}}^{-1}\left\{\mathcal{F}\left\{u_{l-1}{f}_{l-1}\right\}\left(k_x,k_y\right)e^{-j\left(k-\sqrt{k^2-k_x^2-k_y^2}\right)\mathrm{\Delta }z}\right\}.$$ [2]

Here, $u_l\left(x,y\right)$ is the optical field at layer $l$, $\mathcal{F}$ the Fourier transform, $\mathrm{\Delta }z$ the distance between layers, and $k$ the wavenumber in the medium between layers. Each measurement is a collection of $N_v$ intensity patterns $g_i\left(x,y\right)$, with $i=1,\dots ,N_v$, captured on the detector for each orientation $i$ of the sample. In this work, we assume that the sample is a pure phase object; i.e., $\alpha \left(x,y\right)=0$. This assumption is valid for the glass phantom in the optical domain as well as for a sample composed of copper and silicon in the X-ray domain at 17 keV, where the phase contrast (real part of the refractive index contrast) is about 10 times larger than the absorption contrast (imaginary part of the refractive index contrast).

From the measurements, we produce an approximation $\tilde{\mathbf{f}}$ of the phase pattern ${\phi }_l\left(x,y\right)$ for each layer $l$ in the sample. We use the steepest gradient descent method with a fixed number of iterations $K=8$ to generate the approximant. In what follows, we represent the measurements (consisting of $M$ real pixel values) by a ($M\times 1$) column vector ${\mathbf{g}}_i$ and the discretized object (consisting of $N$ real voxel values) by a ($N\times 1$) column vector $\mathbf{f}$. We then define a cost function $J$ to minimize, consisting simply of a data fidelity term

$$J=\frac{1}{2}\sum _{i=1}^{N_v}{\Vert H_{\text{i}}\left(\mathbf{f}\right)-{\mathbf{g}}_i\Vert }_2^2,$$ [3]

where $H_{\text{i}}$ denotes the forward operator that maps the object function $\mathbf{f}$ to a prediction of the measurement $H_{\text{i}}\left(\mathbf{f}\right)$, for a particular orientation $i$ of the sample. In the problem presented here, the optical field will first propagate through the sample $L$ layers, each of thickness $\mathrm{\Delta }z$, and then in free space to the detector over a distance $d$. The forward operator can thus be written as a cascade of Fresnel propagation operations and thin mask multiplications corresponding to the object layers, i.e., successive applications of Eq. 2 written in operator form

$$H_{\text{i}}\left(\mathbf{f}\right)={\left|{\mathbf{u}}_{det}\right|}^2$$ [4]

$${\mathbf{u}}_{det}=F_d\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_L\right]\dots F_{\mathrm{\Delta }z}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_2\right]F_{\mathrm{\Delta }z}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_1\right]{\mathbf{u}}_{\mathrm{i}\mathrm{n}\mathrm{c},i},$$ [5]

where ${\mathbf{f}}_l$ is the vector of object function values in layer $l$, $F_{\mathrm{\Delta }z}$ the Fresnel propagation operator over distance $\mathrm{\Delta }z$, ${\mathbf{u}}_{\mathrm{i}\mathrm{n}\mathrm{c},i}$ the incident field, ${\mathbf{u}}_{det}$ the field on the detector, and $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[\mathbf{v}\right]$ the diagonal matrix with vector $\mathbf{v}$ on the diagonal. The gradient descent iterative update can be written as

$${\mathbf{f}}^{\left(k+1\right)}={\mathbf{f}}^{\left(k\right)}-s{\left({\nabla }_{{\mathbf{f}}^{\left(k\right)}}J\right)}^T,$$ [6]

where ${\mathbf{f}}^{\left(k\right)}$ is the object estimate at iteration $k$, $s$ the step size, and ${\nabla }_{{\mathbf{f}}^{\left(k\right)}}J$ the gradient of $J$ with respect to $\mathbf{f}$ evaluated at ${\mathbf{f}}^{\left(k\right)}$. We then set $\tilde{\mathbf{f}}={\mathbf{f}}^{\left(K\right)}$ with $K$ chosen in advance, starting from ${\mathbf{f}}^{\left(0\right)}=\mathbf{0}$. The detailed derivation of the gradient for the particular model in Eqs. 4 and 5 is given after the conclusion in the Derivation of the Gradient section. In Fig. 3, we give an example of 1 experimental intensity measurement (${\mathbf{g}}_1$) taken from the series of tomographic projections and the corresponding approximant ${\mathbf{f}}^{\left(8\right)}$ obtained from the whole series using Eq. 6.

Fig. 3.

(A) Examples of experimental intensity measurement for the sample orientation $\left({\theta }_x=-10^{○},{\theta }_y=0^{○}\right)$. (B) Phase approximant for IC layer 1 obtained from the collection of 22 intensity patterns at different orientations $\left({\theta }_x=-10^{○},-8^{○},\dots ,+10^{○},{\theta }_y=0^{○}\right)$ and $\left({\theta }_x=0^{○},{\theta }_y=-10^{○},-8^{○},\dots ,+10^{○}\right)$.

DNN Architecture and Training

We use a DNN to map the approximant to the final reconstruction $\widehat{\mathbf{f}}$. The DNN is a convolutional neural network with a DenseNet architecture (62). The implementation is the same as the DNN used in ref. 45 except that the number of dense blocks was reduced to 3 in both the encoder and the decoder, as we empirically observed that using more dense blocks did not result in a significant improvement of the results. We produce a total of 5,500 synthetic sets of measurements obtained by simulating the optical apparatus with the beam propagation method in Eq. 2. The synthetic measurements were subject to simulated shot noise and read noise equivalent to the noise levels found in the experiment. The shot noise was accounted for by converting the simulated measurement pixel intensities $I$ expressed in average photon count per pixel per integration period of the detector to integer numbers of photons $N$ following Poisson statistics. The actual optical power on the camera was measured with a power meter and converted to an average photon flux per detector pixel. Read noise following Gaussian statistics was subsequently added. The parameters (variance and average) of the noise were measured from a series of dark frames from the camera taken with the same gain (EM gain of 1) and integration time (2 ms) as the experimental measurements.

From each set of measurements, we produce a multilayer approximant using the gradient descent in Eq. 6. The examples are split in a training set of 5,000 examples, a validation set of 450 examples, and a test set of 50 examples. Each set of measurements (1 example) comprises 22 views corresponding to different orientations of the sample. The DNN is then trained to map the approximant to the ground truth used for the simulation. Each layer of the sample is assigned to a different channel in the DNN. We use the negative Pearson correlation coefficient ($\text{NPCC}=-\text{PCC}$) as loss function and train in 20 epochs with a batch size of 16 examples. For 2 images $A$ and $B$, the PCC is defined as

$$\mathrm{P}\mathrm{C}\mathrm{C}\left(A,B\right)=\frac{{\sum }_i\left(A_i-Ā\right)\left(B_i-\overline{B}\right)}{\sqrt{{\sum }_i{\left(A_i-Ā\right)}^2{\sum }_i{\left(B_i-\overline{B}\right)}^2}},$$ [7]

The PCC (and therefore the NPCC too) is agnostic to scale and offset; i.e., $\mathrm{P}\mathrm{C}\mathrm{C}\left(aA+b,cB+d\right)=\mathrm{P}\mathrm{C}\mathrm{C}\left(A,B\right)$ for $a,b,c,d\in \mathbb{R}$. As a consequence, the DNN, which is trained by minimizing the NPCC, may apply some offset and scaling to the reconstruction. These parameters are not easily predictable; however, for a given DNN they are constant once training is complete, which allows us to correct the reconstructions. Offset and scale are obtained by least-squares linear regression between the DNN output and the ground truth from the synthetic test set examples (not including the experimental example).

Results

The method described in the previous sections was applied to the glass phantom shown in Fig. 1B. The synthetic measurements were subject to Poisson noise resulting from $10^3$ photon flux per detector pixel, equal to the experimental photon flux, and an additive Gaussian noise with a SD of 13 counts. For DNN training, we compared 2 sets of approximants, obtained with $K=1$ and $K=8$ with and without total variation (TV) regularization. In the case $K=1$, the regularization parameter $\alpha$ was set to 0.1 (step size 0.1). We chose a smaller value of 0.04 for the case $K=8$ (step size 0.05) because the proximal operator corresponding to the regularizer is applied at each iteration and its effect tends to accumulate. In the case $K=8$, the particular choice for the number of iterations is an empirical trade-off between computation time and accuracy. The same optimization parameters (step size and number of iterations) were used to compute the approximant of the IC phantom, and the result for each layer is shown in Fig. 4 E–H for $K=1$ and Fig. 4 I–L for $K=8$. The DNN reconstruction results are summarized in Fig. 4 M–P ($K=1$) and Fig. 4 Q–T ($K=8$). The approximant and the DNN reconstruction represent the phase modulation imposed by each layer in the sample. The absolute phase carries no useful information; therefore we are free to offset the reconstructed phase by an arbitrary constant. In the DNN reconstructions in Fig. 4 I–L, the IC patterns (where the phase shift actually occurs) are typically reconstructed with 0 phase due to the rectified linear units (which project all negative values to 0) at the output layer of the DNN. We reassign the phase of the pattern to the nominal phase of −0.33 rad so that it can be visually compared to the ground truths in Fig. 4 M–P. An alternate approach leading to very similar results is to assign a 0 phase to the background.

The DNN reconstructions can be compared to those obtained using learning tomography (LT), a previously demonstrated optical tomography technique (60, 61) based on proximal optimization (FISTA) (63) with TV regularization (64). The role of the TV regularizer is to favor piecewise constant solutions while preserving sharp edges, which is especially well suited for IC patterns. LT was initially designed for holographic measurements and was modified here to work on intensity measurements by computing the gradient for the data fidelity term in Eq. 3. The essential difference in the LT optimization is that a TV filter playing the role of a proximal operator is applied at each iteration on the current solution. The LT reconstructions for the experimental dataset are shown in Fig. 4 A–D. These particular reconstructions were obtained after 30 iterations of gradient descent, a step size of 0.05, a regularization parameter $\alpha =0.04$, and 20 iterations for the TV regularizer at each step. The computation time of the ${\mathbf{f}}^{\left(8\right)}$ approximant is 51 s for $K=8$ (no regularization) and 6 s for $K=1$, including 570 ms for the DNN vs. 212 s for LT on the same processor (see Materials and Methods for hardware details).

In Table 1, we summarize the values of the PCC, which we use to quantify the quality of the reconstructions. The values are given for reconstructions on the synthetic test set (50 examples) and also the reconstruction of the single experimental example. Because the reconstruction quality turns out to depend strongly on the particular layer, we display the value for each of the 4 layers separately. As may be expected, the values for the LT are higher (better reconstructions) than those for the approximant as LT was run for 30 iterations vs. 8 for the approximant and that the latter was not regularized. The DNN reconstructions appear to be the best according to the PCC metric, which shows that, even while starting from a poor approximation, the DNN was able to outperform LT. Note that a direct comparison between the performance of the DNN on the synthetic data and that on the experimental example is not fair because the ground truth is not known in the experiment. The ground truth used for the experimental example is an idealization from the design parameters used to fabricate the sample. We also indicate the values for reconstructions based on the regularized approximant (using the same regularization parameters as in the LT algorithm). In terms of PCC, there is no significant difference from the unregularized case.

Table 1.

PCC, expressed in percentage (i.e., PCC $\times$ 100), of the reconstructions in the test set with respect to the ground truths for the approximant (not regularized) and the DNN reconstructions, labeled “DNN,” obtained from the unregularized approximant

		Approximant		DNN		DNN reg.		LT
K	Layer	Simul.	Exp.	Simul.	Exp.	Simul.	Exp.	Simul.	Exp.
1	1	62 $\pm$ 7	48	99 $\pm$ 0.3	80	99 $\pm$ 0.4	72	91 $\pm$ 2	65
1	2	43 $\pm$ 5	22	97 $\pm$ 1	56	96 $\pm$ 1	45	79 $\pm$ 7	37
1	3	49 $\pm$ 9	41	99 $\pm$ 0.4	77	94 $\pm$ 5	76	89 $\pm$ 3	62
1	4	24 $\pm$ 7	7	95 $\pm$ 1	38	92 $\pm$ 2	42	76 $\pm$ 7	27
8	1	75 $\pm$ 63	63	100 $\pm$ 0.2	75	100 $\pm$ 0.1	76	—	—
8	2	57 $\pm$ 6.5	31	98 $\pm$ 0.7	44	99 $\pm$ 0.4	45	—	—
8	3	62 $\pm$ 6.5	52	99 $\pm$ 0.3	80	99 $\pm$ 0.3	79	—	—
8	4	41 $\pm$ 8.1	12	96 $\pm$ 0.8	48	98 $\pm$ 0.6	43	—	—

We show the 2 cases $K=1$ and $K=8$ for the approximant calculation. The LT solution is obtained with $K=30$ and is indicated on the right. The values for the DNN trained with regularized approximants are labeled “DNN reg.” The uncertainty values indicated correspond to the SD over the 50 examples of the test set. For each case, the values for the synthetic (simulated) and experimental examples are indicated in separated columns “Simul.” and “Exp.,” respectively. No uncertainty is given for the experimental case as it contains only 1 example.

The reconstructions based on regularized approximants are shown in Fig. 5. By comparing these images with the unregularized reconstructions shown in Fig. 4, and also by considering the value of the PCC in Table 1, we conclude that the regularization has little effect, especially on the experimental reconstructions. Moreover, the TV regularization may not operate as a favorable preconditioner for the DNN. The choice of the TV operator as a regularizer is arbitrary and only based on our assumption that the solution should be piecewise constant. In fact, because of the small angle range, the approximants for the different phantom layers are quite similar to each other and the regularization may cancel information that the DNN could use to discriminate between them. Layers 3 and 4 can be said to look visually better with the regularized approximant, but the situation is reversed for layers 1 and 2. Iterating more, i.e., using $K=8$ vs. $K=1$, yields slightly better results as can be expected intuitively, but the improvement is quite minute considering the increased computation time required to perform 7 more iterations.

Fig. 5.

(A–D) Approximants generated from the experimental measurements with $K=1$ and TV regularization with $\alpha =0.1$. (E–H) Approximants obtained with $K=8$ and TV regularization with $\alpha =0.04$. (I–L) Reconstructions from the DNN of each approximant A–D, respectively. (M–P) Reconstructions from the DNN of each approximant E–H, respectively.

In the regularized case $K=1$ only we observed instability in the behavior of the DNN for the regularized approximants. For bipolar input (approximant containing both positive and negative values), one of the phantom layers (layer 3) would always be reconstructed to null values. As we are using a rectified linear unit as an activation function, this means that the output of one layer within the network displays only negative values. By offsetting the input to the DNN (approximant) so that all values are positive we were able to remove the problem (reconstructions of Fig. 5 I–L). For the regularized $K=1$ case where this behavior was observed, the difference between approximant layers is the smallest; i.e., the failure may be due to the regularizer washing out the differences.

So far, we have reconstructed the phase shift distribution $\phi \left(x,y\right)$ associated to each layer. In fact, it is possible, with the same method, to infer the refractive index $n\left(x,y\right)$ of the sample. For a given layer, the refractive index is simply given by $n\left(x,y\right)=\phi \left(x,y\right)/\left(k\mathrm{\Delta }z\right)$, where $\mathrm{\Delta }z$ is the thickness of the layer. If the layer thicknesses are not known, one would instead slice the object into layers at finer spacing to meet the applicable Nyquist criterion.

Conclusion

In this paper, we have demonstrated through an emulated X-ray experiment that DNNs can be used to improve the reconstruction of IC layouts from tomographic intensity measurements acquired within an angle range of $\pm 10^{○}$ along each lateral axis. The approximant obtained after 1 or several iterations of the steepest gradient method does not provide a reconstruction of sufficient quality for the purpose of IC integrity assessment. The DNN, however, exploits the strong prior contained in the object geometry and yields reconstructions that are significantly improved over the approximant. One of the main motivations for using DNN is indeed the speed of execution; therefore, we want to limit any unnecessary preprocessing. In fact, trying to improve the quality of the approximant by simply iterating more the gradient descent does not yield significant improvement.

One significant challenge for the method we demonstrated is to provide a proper training set for the DNN. In the case of ICs, the training set is simply given by the many real layouts that are available. For more generic objects, a problem needs first to be formulated to clearly define the class of object on which the training will be performed. This is, however, the case in general for problem solving that involves DNNs. What has been shown here is the compelling improvement that DNN can bring to a phase tomography problem when the class of object is known.

More generally, there is a trade-off between the specificity of the required priors and the “difficulty” of an inverse problem—measured as degree of ill-posedness, e.g., the ratio of largest to smallest eigenvalue in a linearized representation of the forward operator. The problem we addressed here is severely ill-posed due to the presence of strong scattering within the object and the limited range and number of angular measurements we allowed ourselves to collect. Therefore, the rather restricted nature of ICs as a class prior is justified; while, at the same time, our approach is addressing an indisputably important practical problem. Detailed determination of the relationship between the degree of ill-posedness and the complexity of the object class prior would be a worthwhile topic for future work.

Derivation of the Gradient.

This derivation follows a path similar to the derivation given in ref. 61. We start from Eq. 3:

$$J=\frac{1}{2}\sum _{i=1}^{N_v}{\Vert H_i\left(\mathbf{f}\right)-{\mathbf{g}}_i\Vert }_2^2$$ [8]

$$=\frac{1}{2}\sum _{i=1}^{N_v}\left(H_i{\left(\mathbf{f}\right)}^T{H}_i\left(\mathbf{f}\right)-2H_i{\left(\mathbf{f}\right)}^T{\mathbf{g}}_i+{\mathbf{g}}_i^T{\mathbf{g}}_i\right).$$ [9]

The gradient of $J$ is defined as

$${\nabla }_{\mathbf{f}}J={\left[\frac{\partial J}{\partial {\phi }_1},\dots ,\frac{\partial J}{\partial {\phi }_N},\frac{\partial J}{\partial {\alpha }_1},\dots ,\frac{\partial J}{\partial {\alpha }_N}\right]}_{{\mathbf{f}}^{\left(k\right)}},$$ [10]

where the object function is defined as $\mathbf{f}=\exp \left[j\mathit{\phi }-\mathit{\alpha }\right]$, with $\mathit{\phi }$ representing the phase delay and $\mathit{\alpha }$ the absorption. In what follows we denote the gradient by $\nabla$ for notational simplicity. We take the gradient of Eq. 9 and, by linearity of the derivation operation and denoting $H_i\left(\mathbf{f}\right)$ by ${\mathbf{H}}_i={\left(H_1,\dots ,H_M\right)}^T$, we get

$$\nabla J=\frac{1}{2}\sum _{i=1}^{N_v}\left[\nabla \left({\mathbf{H}}_i^T{\mathbf{H}}_i\right)-2\nabla \left({\mathbf{H}}_i^T{\mathbf{g}}_i\right)\right].$$ [11]

The term $\nabla \left({\mathbf{g}}_i^T{\mathbf{g}}_i\right)$ is absent because measurements ${\mathbf{g}}_i$ do not depend on the estimate $\mathbf{f}$. Then, by the definition

$$\nabla \mathbf{H}\hspace{0.17em}=\left[\begin{array}{cccccc}\hfill \frac{\partial H_1}{\partial {\phi }_1}\hspace{1em}\hfill & \hfill \dots \hspace{1em}\hfill & \hfill \frac{\partial H_1}{\partial {\phi }_N}\hspace{1em}\hfill & \hfill \frac{\partial H_1}{\partial {\alpha }_1}\hspace{1em}\hfill & \hfill \dots \hspace{1em}\hfill & \hfill \frac{\partial H_1}{\partial {\alpha }_N}\\ \hfill \dots \hspace{1em}\hfill \\ \hfill \frac{\partial H_M}{\partial {\phi }_1}\hspace{1em}\hfill & \hfill \dots \hspace{1em}\hfill & \hfill \frac{\partial H_M}{\partial {\phi }_N}\hspace{1em}\hfill & \hfill \frac{\partial H_M}{\partial {\alpha }_1}\hspace{1em}\hfill & \hfill \dots \hspace{1em}\hfill & \hfill \frac{\partial H_M}{\partial {\alpha }_N}\end{array}\right],$$ [12]

we get

$$\nabla J=\sum _{i=1}^{N_v}\left[{\mathbf{H}}_i^T\nabla {\mathbf{H}}_i-{\mathbf{g}}_i^T\nabla {\mathbf{H}}_i\right]$$ [13]

$$=\sum _{i=1}^{N_v}\left[{\mathbf{r}}_i^T\nabla {\mathbf{H}}_i\right],$$ [14]

where ${\mathbf{r}}_i$ is the residual defined as ${\mathbf{r}}_i={\mathbf{H}}_i-{\mathbf{g}}_i$. Finally, we get the expression required in Eq. 6:

$${\left(\nabla J\right)}^T=\sum _{i=1}^{N_v}{\left(\nabla {\mathbf{H}}_i\right)}^T{\mathbf{r}}_i.$$ [15]

In Eq. 15, ${\left(\nabla {\mathbf{H}}_i\right)}^T$ is a matrix of size ($2N\times M$) that is too large to be computed directly. Instead, we use a routine, described below, to calculate the vector ${\left(\nabla {\mathbf{H}}_i\right)}^T{\mathbf{r}}_i$ directly. We remind the reader of Eqs. 4 and 5 that describe the forward model where we drop the index $i$ to simplify the notation as the expression assumes the same form for each sample orientation:

$$\mathbf{H}={\left|{\mathbf{u}}_{det}\right|}^2$$ [16]

$${\mathbf{u}}_{det}=F_d\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_L\right]\dots F_{\mathrm{\Delta }z}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_2\right]F_{\mathrm{\Delta }z}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_1\right]{\mathbf{u}}_{\mathrm{i}\mathrm{n}\mathrm{c}}.$$ [17]

This forward operator allows for a convenient computation of the gradient by using a backpropagation scheme. We first calculate the gradient of Eq. 16,

$$\nabla \mathbf{H}=\nabla |{\mathbf{u}}_{det}{|}^2$$ [18]

$$=\nabla \left(\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_{det}^{*}\right]{\mathbf{u}}_{det}\right)$$ [19]

$$=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_{det}\right]\nabla {\mathbf{u}}_{det}^{*}+\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_{det}^{*}\right]\nabla {\mathbf{u}}_{det}$$ [20]

$$=2\mathfrak{R}\left\{\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_{det}^{*}\right]\nabla {\mathbf{u}}_{det}\right\},$$ [21]

where the asterisk represents the complex conjugate. Thus,

$${\left(\nabla \mathbf{H}\right)}^{†}\mathbf{r}=2\mathfrak{R}\left\{{\left(\nabla {\mathbf{u}}_{det}\right)}^{†}\mathbf{r}\prime \right\},$$ [22]

where the dagger represents the Hermitian transpose and we have defined $\mathbf{r}\prime =\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_{det}\right]\mathbf{r}$. Because it is not practical to compute the matrix ${\left(\nabla {\mathbf{u}}_{det}\right)}^{†}$ due to its size, we use a recursive scheme to compute ${\left(\nabla {\mathbf{u}}_{det}\right)}^{†}\mathbf{r}\prime$ directly. For that, we rewrite Eq. 17 as a recursive relationship for the optical field ${\mathbf{u}}_l$ just after layer $l$:

$${\mathbf{u}}_1=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_1\right]{\mathbf{u}}_{\mathrm{i}\mathrm{n}\mathrm{c}}$$ [23]

$${\mathbf{u}}_l=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_l\right]F_{\mathrm{\Delta }z}{\mathbf{u}}_{l-1}$$ [24]

$${\mathbf{u}}_{det}=F_d{\mathbf{u}}_L.$$ [25]

The optical field $\mathbf{u}$ is thus known everywhere for a given object function $\mathbf{f}$. We then propagate the residual $\mathbf{r}\prime$ backward through the sample by using the same propagation relationships:

$${\mathbf{r}}_L^{\prime }=F_d^{†}\mathbf{r}\prime$$ [26]

$${\mathbf{r}}_{l-1}^{\prime }=F_{\mathrm{\Delta }z}^{†}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[{\mathbf{f}}_l\right]}^{†}{\mathbf{r}}_l^{\prime }.$$ [27]

Note that the Fresnel operator is unitary; i.e., $F^{†}=F^{-1}$. We take the gradient of Eqs. 23–25:

$$\nabla {\mathbf{u}}_1=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_{\mathrm{i}\mathrm{n}\mathrm{c}}\right]\nabla {\mathbf{f}}_1$$ [28]

$$\nabla {\mathbf{u}}_l=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[F_{\mathrm{\Delta }z}{\mathbf{u}}_{l-1}\right]\nabla {\mathbf{f}}_l+\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{f}}_l\right]F_{\mathrm{\Delta }z}\nabla {\mathbf{u}}_{l-1}$$ [29]

$$\nabla {\mathbf{u}}_{det}=F_d\nabla {\mathbf{u}}_L.$$ [30]

We then take the Hermitian transpose and multiply by the residual; we get

$${\left(\nabla {\mathbf{u}}_1\right)}^{†}{\mathbf{r}}_1^{\prime }={\left(\nabla {\mathbf{f}}_1\right)}^{†}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[{\mathbf{u}}_{\mathrm{i}\mathrm{n}\mathrm{c}}\right]}^{†}{\mathbf{r}}_1^{\prime }$$ [31]

$$\begin{array}{ll}\hfill {\left(\nabla {\mathbf{u}}_l\right)}^{†}{\mathbf{r}}_l^{\prime }& ={\left(\nabla {\mathbf{f}}_l\right)}^{†}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[F_{\mathrm{\Delta }z}{\mathbf{u}}_{l-1}\right]}^{†}{\mathbf{r}}_l^{\prime }\hspace{0.17em}+\hfill \\ \hfill & \hspace{1em}+{\left(\nabla {\mathbf{u}}_{l-1}\right)}^{†}{F}_{\mathrm{\Delta }z}^{†}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[{\mathbf{f}}_l\right]}^{†}{\mathbf{r}}_l^{\prime }\hfill \end{array}$$ [32]

$${\left(\nabla {\mathbf{u}}_{det}\right)}^{†}\mathbf{r}\prime ={\left(\nabla {\mathbf{u}}_L\right)}^{†}{F}_d^{†}\mathbf{r}\prime .$$ [33]

We simplify the equations above by making use of Eqs. 24 and 25,

$${\left(\nabla {\mathbf{u}}_1\right)}^{†}{\mathbf{r}}_1^{\prime }={\left(\nabla {\mathbf{f}}_1\right)}^{†}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[{\mathbf{u}}_{\mathrm{i}\mathrm{n}\mathrm{c}}\right]}^{†}{\mathbf{r}}_1^{\prime }$$ [34]

$${\left(\nabla {\mathbf{u}}_l\right)}^{†}{\mathbf{r}}_l^{\prime }={\left(\nabla {\mathbf{u}}_{l-1}\right)}^{†}{\mathbf{r}}_{l-1}^{\prime }+{\left(\nabla {\mathbf{f}}_l\right)}^{†}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[F_{\mathrm{\Delta }z}{\mathbf{u}}_{l-1}\right]}^{†}{\mathbf{r}}_l^{\prime }$$ [35]

$${\left(\nabla {\mathbf{u}}_{det}\right)}^{†}\mathbf{r}\prime ={\left(\nabla {\mathbf{u}}_L\right)}^{†}{\mathbf{r}}_L^{\prime },$$ [36]

which gives us a recursive relationship to calculate the gradient of the field at each layer. Note that ${\left(\nabla {\mathbf{f}}_l\right)}^{†}$ is a matrix of size $2N\times M$ whose entries are nonzeros only for the diagonal entries corresponding to layer $l$ because $f_i$ depends only on ${\alpha }_i$ and ${\phi }_i$. In practice, ${\left(\nabla {\mathbf{u}}_{det}\right)}^{†}\mathbf{r}\prime$ can be built layer by layer by stacking the second term of the right-hand side of Eq. 35 which reads, for pure phase objects ($\mathit{\alpha }=\mathbf{0}$),

$${\left(\nabla {\mathbf{f}}_l\right)}^{†}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[F_{\mathrm{\Delta }z}{\mathbf{u}}_{l-1}\right]}^{†}{\mathbf{r}}_l^{\prime }=-j\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[e^{-j{\mathit{\phi }}_l}\right]\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left[F_{\mathrm{\Delta }z}{\mathbf{u}}_{l-1}\right]}^{†}{\mathbf{r}}_l^{\prime }$$ [37]

$$=-j\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[e^{-j{\mathit{\phi }}_l}\right]\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[e^{j{\mathit{\phi }}_l}\right]\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_l^{*}\right]{\mathbf{r}}_l^{\prime }$$ [38]

$$=-j\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_l^{*}\right]{\mathbf{r}}_l^{\prime },$$ [39]

where we have used Eq. 24. Finally, according to Eq. 22, we obtain layer $l$ of ${\left(\nabla \mathbf{H}\right)}^{†}\mathbf{r}$,

$${\left(\nabla \mathbf{H}\right)}^{†}\mathbf{r}{\mid }_{\mathrm{l}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\hspace{0.17em}l}=2\mathfrak{I}\left\{\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{u}}_l^{*}\right]{\mathbf{r}}_l^{\prime }\right\},$$ [40]

where $\mathfrak{I}$ denotes the imaginary part.

Materials and Methods

The experimental apparatus is shown in Fig. 2A. The light source is a CW He-Ne laser at $632.8\hspace{0.17em}\mathrm{n}\mathrm{m}$ that is spatially filtered, expanded, and collimated into a quasi-plane wave with an Airy disk intensity profile of 33 mm in diameter. The sample is mounted on a 2-axis rotation stage rotating along the x and y axes. The sample middle plane is imaged using a demagnifying telescope ($\times 0.50$) lens system to enhance the effect of diffraction and increase the field of view on the detector. The detector is an EM-CCD (QImaging Rolera EM-C2) with a 1,004 $\times$ 1,002 array of $8\times 8$-μm pixels. To simulate the diffraction occurring in an X-ray measurement, the detector is defocused by a distance $\mathrm{\Delta }z=58$ mm from the image plane, which corresponds to Fresnel numbers ranging from 0.7 to 5.5 for the different object features.

The sample corresponds to a $10^4\times$ scale-up of a real IC design. The original IC comprises 13 layers, including the doped layers. We kept only layers 5 to 8 from the original design (relabeled here 1 to 4) shown in Fig. 2C that contain copper patterns that would induce a significant phase delay in the X-ray regime. The 4 glass plates corresponding to the IC layers were cut in a 500-μm-thick fused silica wafer and $575\pm 5$-nm deep patterns (measured after fabrication with a Bruker DekTak XT stylus profilometer) were obtained by wet etching. To control the phase contrast and reduce parasitic reflections between the layers, we used an immersion oil (Fig. 2B) with a refractive index $n_D=1.400\pm 0.0002$ at 25 °C from Cargille-Sacher Laboratories. According to the manufacturer, the refractive index of the oil is $n_{\mathrm{o}\mathrm{i}\mathrm{l}}=1.4005\pm 0.0002$ at $632.8\hspace{0.17em}\mathrm{n}\mathrm{m}$ and 20 °C. The refractive index of fused silica is $n_{\mathrm{g}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}}=1.457$ at $632.8\hspace{0.17em}\mathrm{n}\mathrm{m}$ and 20°C (65), which gives a contrast of $\mathrm{\Delta }n=0.0565\pm 0.0005$. The corresponding phase shift for a single pattern is then $\mathrm{\Delta }\phi =kd\mathrm{\Delta }n=0.323\pm 0.006$ rad.

The sample layers are fabricated on double-sided polished 150-mm-diameter and 500-μm-thick fused silica wafers. A 1-μm-thick positive tone resist (Megaposit SPR700) is spin coated at 3,500 rpm on both sides of the wafer and soft baked at 95 °C for 30 min in a convection oven. The backside was also coated for protection from the forthcoming wet etch. Scaled versions of IC designs in GDSII format are then patterned directly using a maskless aligner (MLA150; Heidelberg Instruments) with a 405-nm laser and developed using an alkaline developer (Shipley Microposit MF CD-26) for 45 s followed by a deionized (DI) water rinse and ${\mathrm{N}}_2$ drying. A hard bake at 120 °C for 30 min is carried out to stabilize the patterned features. A short descum of 2 min at 1,000-W and 0.1-Torr ${\mathrm{O}}_2$ pressure in a barrel asher is also performed to remove any resist residue. The wafers are subsequently etched for 7 min at a rate of $\sim 80\hspace{0.17em}\mathrm{n}\mathrm{m}$/min in buffered oxide etch. The resist is then stripped from the wafer by a long ash (1 h) followed by a Piranha clean (3:1 ${\mathrm{H}}_2{\mathrm{S}\mathrm{O}}_4$:${\mathrm{H}}_2{\mathrm{O}}_2$), DI water rinse, and ${\mathrm{N}}_2$ drying. Finally, the wafers are diced into squares of $50\hspace{0.17em}\mathrm{m}\mathrm{m}\times$ $50\hspace{0.17em}\mathrm{m}\mathrm{m}$ and cleaned again with Piranha, DI water rinse, and ${\mathrm{N}}_2$ drying.

The computation of the approximant is performed with the MATLAB software on an Intel i9-7900X processor running at 3.3 GHz. The DNN training and testing are performed under Keras with Tensorflow backend running on an NVIDIA Titan Xp graphics processing unit.