Learning hidden elasticity with deep neural networks

Significance

Elastography has been applied to medical imaging to detect cancerous tissues for decades. The conventional elastography based on the assumption of stress uniformity is prone to inaccuracies. Accurately reconstructing elasticity distributions of heterogeneous objects requires solving an inverse problem in elasticity, which is difficult due to its ill-posed nature. By implementing machine learning techniques, we have developed ElastNet, an elastography method for extracting the hidden elasticity of objects based on measured strain distributions. We show that by combining the theory of elasticity with a deep-learning approach, ElastNet can provide rapid and accurate predictions. We validate ElastNet by applying it to various elastography problems with complex elasticity distributions.

Abstract

Elastography is an imaging technique to reconstruct elasticity distributions of heterogeneous objects. Since cancerous tissues are stiffer than healthy ones, for decades, elastography has been applied to medical imaging for noninvasive cancer diagnosis. Although the conventional strain-based elastography has been deployed on ultrasound diagnostic-imaging devices, the results are prone to inaccuracies. Model-based elastography, which reconstructs elasticity distributions by solving an inverse problem in elasticity, may provide more accurate results but is often unreliable in practice due to the ill-posed nature of the inverse problem. We introduce ElastNet, a de novo elastography method combining the theory of elasticity with a deep-learning approach. With prior knowledge from the laws of physics, ElastNet can escape the performance ceiling imposed by labeled data. ElastNet uses backpropagation to learn the hidden elasticity of objects, resulting in rapid and accurate predictions. We show that ElastNet is robust when dealing with noisy or missing measurements. Moreover, it can learn probable elasticity distributions for areas even without measurements and generate elasticity images of arbitrary resolution. When both strain and elasticity distributions are given, the hidden physics in elasticity—the conditions for equilibrium—can be learned by ElastNet.

Being able to reconstruct mechanical property distributions of objects noninvasively has a broad range of applications in materials science (1), civil engineering (2), biomedical engineering (3), and clinical diagnosis (4, 5). As various diseases progress, the stiffness (elasticity) of cells, tissues, and organs are often altered (6). Palpation, a self-screening procedure for tumors, utilizes the difference in elasticity between healthy and cancerous tissues to distinguish them. The displacement (or strain) distribution of an elastic body under deformation can be acquired by a variety of imaging techniques (e.g., ultrasound, MRI, digital image correlation) (1, 3–5). If the stress distribution of the body is also known, the elasticity distribution can be calculated by the elastic constitutive relation (Hooke’s law). However, there is currently no technique that can measure the stress distribution of a body in vivo. Therefore, in the applications of elasticity imagining (elastography), the stress distribution of a body is commonly assumed to be uniform, and the measured strain distribution is interpreted as a relative elasticity distribution. This approach is referred to as strain-based elastography and has the advantage of being easy to implement (7). The assumption of stress uniformity, however, is inaccurate. The stress field of a body can be distorted significantly near a hole, inclusion, or wherever the elasticity varies. This phenomenon, known as stress concentration, is of great interest in industry and academia (8–10). To mitigate this misinterpretation, a research field focusing on solving an inverse problem associated with elastography has been extensively investigated for decades (11–13). In this approach, referred to as model-based elastography, the elasticity distribution of a body can be, in principle, reconstructed by modeling the elastic behavior of the body and solving the inverse problem in elasticity.

To tackle this inverse problem, the predominant approaches in the literature are based on minimizing the difference between the measured and simulated displacements (7, 14–17). These approaches are computationally expensive as they need many iterations—each iteration requires solving a forward problem (e.g., using the finite element analysis) and conducting sensitivity analysis in order to update the prediction. It is possible to solve this inverse problem directly. In other approaches, known as direct approaches, measurements are considered as coefficients in the partial differential equations (PDEs) for equilibrium ($\nabla \mathit{\sigma }=0$) (18–20). While the direct approaches are computationally more efficient than the iterative approaches, they often perform poorly when the measurements contain large strain gradients or noise or when the elasticity distribution is not smooth (not differentiable). Moreover, the error from noise tends to propagate along the integration path when solving the PDEs, thus producing low-quality results (18). Due to these limitations, model-based elastography methods were mostly applied to simple elastography problems such as a uniform body containing a few circular inclusions.

Inverse problems arise in many scientific and engineering fields and are difficult to solve using conventional approaches (21–23). In recent years, much progress toward artificial intelligence and machine learning (ML) has been made and provided novel directions to solve inverse problems. For instance, ML techniques were applied to inverse problems in materials design (24–26), fluid mechanics (27, 28), and many others (29–34). To obtain useful information from an elasticity image (e.g., to identify potential tumors), the number of pixels (resolution) is typically on the order of 10³ to 10⁵. Due to the high-dimensional input space (strain distribution) and output space (elasticity distribution), the hidden correlation between them is difficult to capture by conventional supervised learning using labeled data. In principle, supervised learning may still work for this inverse problem if the number of possible elasticity distributions can be largely reduced. A simple way to do so, for instance, is to consider a uniform body containing a few inclusions and constrain the possible shapes, sizes, locations, and elastic moduli of these inclusions (35). However, when an ML model is trained in this manner, adding such artificial constraints limits the application of the model in practice.

Results

Deep Learning for Elastography.

We introduce ElastNet, a de novo elastography method to learn the hidden elasticity of a body with a deep neural network (DNN). The framework of ElastNet is summarized in Fig. 1. The DNN $f_{\theta }$ with learning parameters $\theta$ is not trained from labeled data—it is never shown the correct elasticity distribution. By contrast, it is supervised by the theory of elasticity, thus allowing it to escape the performance ceiling imposed by labeled data. The elastic constitutive relation ($\mathit{\sigma }=\mathit{C\epsilon }$) and equilibrium equation ($\nabla \mathit{\sigma }=0$) for solving the inverse problem in elasticity are encoded in the DNN as prior knowledge. Biological tissues are mainly composed of water, and they are nearly incompressible (4). Here, all material points in a body of interest are assumed to be linear, isotropic, and incompressible. The elasticity at a material point can be described by either Young’s modulus $E$ or shear modulus $G$. Without loss of generality, Young’s modulus $E$ is used to quantify the elasticity in this work. The body is assumed to be a thin plate, and thus the nonzero stress components are ${\sigma }_{xx}$, ${\sigma }_{yy}$, and ${\tau }_{xy}$ (plane stress state). The information of position $p$ and strain $\epsilon$ at all material points is converted to a data set. The stress $\sigma$ at each material point is calculated by the encoded elastic constitutive relation based on the measured strain $\epsilon$ and predicted elasticity $E$ (see Materials and Methods). The DNN $f_{\theta }$ takes the information of position $p$ at each material point as an input and outputs its elasticity, $E=f_{\theta }\left(p\right)$. The information of strain $\epsilon$ at each material point is not involved in the forward-propagation for predicting the elasticity but is used in the backpropagation for updating the weights $\theta$. When the entire data set is passed forward through the DNN, a predicted stress distribution $\mathit{\sigma }$ is generated.

Fig. 1.

Framework of ElastNet. The DNN $f_{\theta }$ takes the information of position $p$ at each material point as an input and outputs its elasticity, $E=f_{\theta }\left(p\right)$. The stress $\sigma$ at each material point is calculated by the encoded elastic constitutive relation ($\mathit{\sigma }=\mathit{C\epsilon }$) based on the measured strain $\epsilon$ and predicted elasticity $E$. When the entire data set is passed forward through the DNN, a predicted stress distribution $\mathit{\sigma }$ is generated. Residual force maps are then generated by the convolution operation. The training procedure minimizes the norms of the residual forces with an additional physical constraint and updates the weights $\theta$ using backpropagation. Consequently, the predicted elasticity distribution $\mathit{E}$ is updated and then used in the next iteration of training. This training procedure repeats until the predicted elasticity distribution $\mathit{E}$ is converged.

A predicted stress distribution $\mathit{\sigma }$ includes three stress images (${\sigma }_{xx},{\sigma }_{yy},\hspace{0.17em}{\tau }_{xy}$). Before training, these strain images are unlikely to satisfy the conditions for equilibrium, as the initial elasticity distribution $\mathit{E}$ is generated by the DNN with random weights $\theta$. To evaluate how close the predicted stress distribution $\mathit{\sigma }$ is from equilibrium, the stress images are passed forward through a convolutional layer in the DNN. Unlike other convolutional neural networks, in which the kernels need to be learned from labeled data, the kernels in our convolutional layer are encoded in such a way that the convolution operation is used to evaluate the conditions for equilibrium (explained in Learning the Conditions for Equilibrium). Unbalanced (residual) force maps are then generated by the convolution operation. The training procedure minimizes the norms of the residual forces with an additional physical constraint (to be discussed) and updates the weights $\theta$ using backpropagation. Consequently, the predicted elasticity distribution $\mathit{E}$ is updated and then used in the next iteration of training. This training procedure repeats until the predicted elasticity distribution $\mathit{E}$ is converged.

Learning the Conditions for Equilibrium.

The deep-learning framework in ElastNet is similar in spirit to the so-called physics-informed deep learning (28, 29, 31), in which physical laws are encoded into the loss function. Most physics-informed models used automatic differentiation (36) to solve the PDEs in physics. However, automatic differentiation may not be appropriate for the inverse problem in this work. As mentioned, measured strains naturally contain noise, and differentiating the strains can amplify the noise significantly. Moreover, this approach requires the elasticity distribution of a body to be differentiable, which is often not true in practice. Instead of using automatic differentiation, ElastNet uses a convolution operation to solve the PDEs for equilibrium. To elaborate on the idea, we consider a small cube with sides of length $h$. The equilibrium conditions for the cube can be expressed in finite difference forms as (see Materials and Methods):

$${\sigma }_{xx}\left(x+h,y\right)-{\sigma }_{xx}\left(x,y\right)+\hspace{0.17em}{\tau }_{yx}\left(x,y+h\right)-{\tau }_{yx}(x,y)=0,$$ [1]

$${\sigma }_{yy}\left(x,y+h\right)-{\sigma }_{yy}\left(x,y\right)+\hspace{0.17em}{\tau }_{xy}\left(x+h,y\right)-{\tau }_{xy}(x,y)=0.$$

We assume that the cube is made up of $3\times 3$ material points as shown in Fig. 2A. Eq. 1 can be expressed in terms of the stress components at these material points:

$${\displaystyle \sum }_{a=1}^3{\displaystyle \sum }_{b=1}^3{w}_{xx}(a,b){\sigma }_{xx}(a,b)+w_{yy}(a,b){\sigma }_{yy}(a,b)+w_{xy}(a,b){\tau }_{xy}(a,b)=0,$$ [2]

where $w_{xx}$, $w_{yy}$, and $w_{xy}$ are the convolution kernels for ${\sigma }_{xx}$, ${\sigma }_{yy}$, and ${\tau }_{xy}$, respectively. By choosing proper values for the kernels, Eq. 2 can be used to describe the equilibrium conditions for the cube. Two sets of kernels are encoded in the DNN to describe the equilibrium conditions in the x-direction and y-direction, respectively (Fig. 2A). These kernels may therefore be viewed as “equilibrium detectors”. Residual force maps (Fig. 1) can be generated by the convolution operation, in which each residual force is calculated by

$$e(i,j)={\displaystyle \sum }_{a=1}^3{\displaystyle \sum }_{b=1}^3\left(w_{xx}(a,b){\sigma }_{xx}(i+a-1,j+b-1)+w_{yy}(a,b){\sigma }_{yy}(i+a-1,j+b-1)+w_{xy}(a,b){\tau }_{xy}(i+a-1,j+b-1)\right)ht,$$ [3]

where $t$ is the thickness of the cube. Here, we show that the conditions for equilibrium can be encoded in the DNN as domain knowledge for solving the inverse problem. Now, we consider the possibility of learning this domain knowledge from labeled data. To test this idea, an inhomogeneous body is modeled by the finite element method (FEM) with a $128\times 128$ mesh. The elasticity field of the body is defined by a two-dimensional sinusoidal function, $E\left(x,y\right)=0.45\left(1+\cos \left(\frac{4\pi x}{L}\right)\cos \left(\frac{4\pi y}{L}\right)\right)+0.1$. The unit is megapascal (MPa). Thus, the maximum and minimum modulus values are $1$ MPa and $0.1$ MPa, respectively. This “sinusoidal” model is subjected to externally applied displacements along the x-direction on the boundary. An average normal strain (${\epsilon }_{xx}$) of 1% is introduced by the applied displacements. The details of the finite element analysis can be found in our previous work (26) and are summarized in Materials and Methods.

Fig. 2.

Encoding and learning equilibrium conditions. (A) A representative cube consists of $3\times 3$ material points. The equilibrium conditions for the cube can be expressed in terms of the stress components at these material points. Two sets of kernels are encoded in the DNN to describe the equilibrium conditions in the x-direction (upper three kernels) and y-direction (lower three kernels), respectively. (B) The sinusoidal model is subjected to externally applied displacements along the x-direction on the boundary. An average normal strain (${\epsilon }_{xx}$) of $1\%$ is introduced by the applied displacements. To learn the conditions for equilibrium, the elasticity and strain distributions are both fed into the DNN, and the loss function is defined as the MAE of the residual forces. The kernels learned by the DNN are almost identical to those derived mathematically shown in A.

The elasticity and strain distributions of the sinusoidal model are shown in Fig. 2B. To learn the conditions for equilibrium, the elasticity and strain distributions are both fed into the DNN, and the loss function is defined as the mean absolute error (MAE) of the residual forces. From Eq. 3, it can be seen that the kernels ($w_{xx}$, $w_{yy}$, and $w_{xy}$) cannot be uniquely determined by minimizing the norms of the residual forces. A trivial solution is to set all of the kernel values to zeros. Thus, to obtain physically meaningful kernels, additional information must be given. For instance, when the kernel for ${\tau }_{xy}$ to describe the equilibrium condition in the x-direction is given, the other two kernels, which are for ${\sigma }_{xx}$ and ${\sigma }_{yy}$, can be learned. Similarly, when the kernel for ${\tau }_{xy}$ to describe the equilibrium condition in the y-direction is given, the other two kernels can be learned. The kernels learned by the DNN are shown in Fig. 2B and are almost identical to those derived mathematically shown in Fig. 2A. The results show that the kernels encoded in the DNN to describe the conditions for equilibrium are correct, as the same kernels can be learned from the hidden correlation between the elasticity and strain distributions by the DNN.

Learning Hidden Elasticity.

A strain distribution alone does not provide sufficient information to generate a unique elasticity distribution; an additional physical constraint must be imposed (12, 19). We first investigate the effect of physical constraints on prediction accuracy (SI Appendix, Fig. S1 and Movies S1 and S2). We demonstrate that ElastNet can generate accurate predictions when imposing a proper physical constraint based on either the total applied force on the boundary or the mean elasticity. In practice, measured strains naturally contain a certain amount of noise. We also investigate the effect of noise in measurements on prediction accuracy (SI Appendix, Fig. S2 and Movies S3 and S4). We show that ElastNet is robust when dealing with noisy measurements. To evaluate the performance of ElastNet, we compare it with OpenQSEI (17), an iterative model-based elastography method using FEM. We show that ElastNet generates more accurate predictions and is computationally more efficient compared with OpenQSEI (SI Appendix, Figs. S3–S6 and Tables S1–S3).

Most elastography problems assume that measurements are known everywhere in a body. Here, we consider a more challenging problem in which some of the measurements in a body are missing. Noisy measurements may still provide useful information for learning the hidden elasticity. However, missing measurements not only provide no information for solving the inverse problem but also may cause the calculation to break down. Here, we consider a uniform body containing a soft inclusion with a shape similar to the University of California, Berkeley “Cal” logo. In this “Cal” model, the modulus values of the body and soft inclusion are $1$ MPa and $0.1$ MPa, respectively. The elasticity and strain distributions of the model are shown in Fig. 3A. In the first scenario, we assume that there is no missing data in the measured strains. In the second scenario, we assume that the measured strains in an arbitrary area are missing (set to zeros). The predicted elasticity distributions and relative error maps (compared with the ground-truth elasticity distribution) are shown in Fig. 3B, and the error over the training epochs is shown in Fig. 3C. In the first scenario, the predicted elasticity distribution is accurate with a mean relative error (MRE) of $3.01\%$. Larger errors occur on the boundary between the body and soft inclusion due to large elasticity differences. In the second scenario, the predicted elasticity distribution is still accurate with an MRE of $6.97\%$, given that the measurements on a squared area (corresponds to $6.25\%$ of the total area) are missing (yellow-boxed area in Fig. 3B). Larger errors occur on the boundary between the body and soft inclusion and also in the area without data. If the area without data is excluded, the MRE is reduced to $2.95\%$, which is almost the same as the MRE observed in the first scenario with the full data ($3.01\%$). The results show that, for ElastNet, missing measurements only reduce the prediction accuracy in the area without data but do not affect the prediction accuracy in the other areas. Moreover, the DNN is trained to learn the hidden elasticity as a function of positions (Fig. 1). Once such a function is learned, ElastNet can predict a probable elasticity distribution for the area even without measurements (inner figure of Fig. 3C). The learning processes for the first and second scenarios are shown in Movies S5 and S6, respectively. Lastly, we compare ElastNet with DeepFill (37, 38), one of the state-of-the-art image inpainting methods. DeepFill is based on a variant of generative adversarial networks (39), named SN-PatchGAN, with gated convolution trained with millions of images. We use DeepFill to fill missing pixels of predicted elasticity images generated by ElastNet and show that the results obtained from both methods are of equivalent quality (SI Appendix, Figs. S7 and S8 and Table S4).

Fig. 3.

Effect of missing measurements. (A) The ground-truth elasticity distribution (in grayscale) and strain images (in color) of the “Cal” model. The unit in the elasticity distribution is MPa and that in the strain images is percentage ($\%$). (B) The predicted elasticity distributions (in grayscale) and relative error maps (in color) for the scenario with the full data and that with missing data. The unit in the error maps is percentage ($\%$). The yellow-boxed area represents the area without data (corresponds to $6.25\%$ of the total area). (C) The blue and red shaded lines represent the MAE over the training epochs for the scenario with the full data and that with missing data, respectively. The DNNs are trained $100$ times with different initial weights. The line and shading represent the median and interquartile range of the $100$ predictions, respectively. The inner figure shows a probable elasticity distribution for the area without data predicted by ElastNet.

Superresolution Elasticity Imaging.

For most elastography methods, the resolution of predicted elasticity distributions (elasticity images) is limited by the resolution of measurements. ElastNet can generate elasticity images of arbitrary resolution. Here, we apply ElastNet to generate a high-resolution elasticity image from low-resolution measurements. Inspired by the beauty and complexity of Mona Lisa by Leonardo da Vinci, we consider an inhomogeneous body with an elasticity distribution similar to the painting. In this “Mona Lisa” model, the maximum and minimum modulus values are $1$ MPa and $0.1$ MPa, respectively. The elasticity and strain distributions of the model are shown in Fig. 4A. The predicted elasticity distribution and relative error map are shown in Fig. 4B, and the error over the training epochs is shown in Fig. 4C. While the Mona Lisa model has an extremely complex elasticity distribution, the prediction accuracy is high with an MRE of $2.73\%$. No visible difference between the ground truth and prediction can be seen. To understand how ElastNet learns the hidden elasticity, intermediate predictions generated during the learning process are shown in the inner figures of Fig. 4C. Interestingly, it can be seen that ElastNet draws an outline first and then adds more details gradually. This process is similar to how an artist draws. The Mona Lisa model is discretized with a $128\times 128$ mesh. Thus, the resolution of the measured strains is $128\times 128$ (Fig. 4A). After learning the hidden elasticity from the measured strains, ElastNet can generate elasticity images of arbitrary resolution. Here, ElastNet is applied to generate an elasticity image of a higher resolution—$512\times 512$. For comparison, a crop of the ground-truth image and that of the superresolution image are shown in Fig. 4D. The superresolution image seems realistic and provides more details compared with the ground-truth image.

Fig. 4.

Superresolution elasticity imaging. (A) The ground-truth elasticity distribution (in grayscale) and strain images (in color) of the Mona Lisa model. The unit in the elasticity distribution is MPa and that in the strain images is percentage ($\%$). (B) The predicted elasticity distribution (in grayscale) and relative error map (in color). The unit in the error maps is percentage ($\%$). (C) The blue shaded line represents the MAE over the training epochs. The DNN is trained $100$ times with different initial weights. The line and shading represent the median and interquartile range of the $100$ predictions, respectively. The inner figures show the intermediate predictions after $280$, $\mathrm{1,200}$, $\mathrm{2,100}$, and $\mathrm{3,000}$ epochs, respectively. (D) A crop of the ground-truth image ($128\times 128$) and that of the superresolution image ($512\times 512$) generated by ElastNet.

A conventional deep-learning model is trained on a data set and can be applied to predict an elasticity distribution based on a new strain distribution without retraining the model (35). ElastNet, on the other hand, is not supervised by labeled data, and thus its performance is not limited by the amount, distribution, and accuracy of the data. However, ElastNet needs to be retrained for different elastography problems. Therefore, the computational efficiency of ElastNet is essential for elasticity imaging in practice. To make an accurate prediction for the Mona Lisa model (Fig. 4B), ElastNet takes about $80$ min for training $\mathrm{800,000}$ epochs on a workstation using a single graphics processing unit (GPU). After about $\mathrm{3,000}$ epochs (∼$20$ s) of training, an intermediate prediction with much detail can already be obtained (inner figures of Fig. 4C). The learning process is shown in Movie S7. With further improvements on the DNN architecture, ElastNet may be applied in an environment requiring real-time elasticity imaging with broad implications in various fields. Lastly, we compare ElastNet with ESRGAN (40), which won first place in the 2018 Perceptual Image Restoration and Manipulation Challenge on Perceptual Image Superresolution. We apply ESRGAN to produce a high-resolution version of a predicted elasticity image generated by ElastNet and show that the results obtained from both methods are comparable (SI Appendix, Fig. S9 and Table S5).

Discussion

Conventional model-based elastography using FEM represents an elasticity image as a set of pixels. In this work, we consider an elasticity image as a mathematical function. Given a large enough DNN, any complex elasticity image can be approximated by the DNN. This approach ensures that a target elasticity distribution can be represented as a function and allows a learning algorithm to gradually update the function. Here, all material points in a body of interest are assumed to be linear, isotropic, and incompressible. However, ElastNet can be extended to consider compressible materials if necessary. For instance, instead of setting Poisson’s ratio at each material point to $0.5$, it can be represented as an unknown variable. Thus, the DNN $f_{\theta }$ will take the information of position $p$ at each material point as an input and output its elasticity and Poisson’s ratio simultaneously, $\left(E,\nu \right)=f_{\theta }\left(p\right)$. We show that, by combining the theory of elasticity with a deep-learning approach, ElastNet can generate rapid and accurate predictions. The prediction accuracy depends on the complexity of the hidden elasticity. A higher prediction accuracy may be expected when the elasticity distribution is simpler or smoother. We also show that ElastNet is robust when dealing with noisy measurements. For measurements with missing data, only the prediction accuracy in the area without data will be compromised and the prediction accuracy in the other areas will not be affected. Once the function of an elastic image is learned, the DNN can predict probable elasticity distributions for areas even without measurements and generate elasticity images of arbitrary resolution. To obtain better performance, incorporating other DNNs specifically trained for image inpainting or single-image superresolution into ElastNet is recommended. With prior knowledge from the theory of elasticity, ElastNet does not require any labeled data for training, and thus no artificial constraint on possible elasticity distributions needs to be imposed. This advantage allows ElastNet to be applied to a broad range of elastography problems in which no prior knowledge is available on the hidden elasticity.

Materials and Methods

Finite Element Analysis.

A body of interest is discretized by four-node quadrilateral elements with a $128\times 128$ mesh. The modulus of each element is determined based on the elasticity distribution of the body. To simulate an incompressible material, Poisson’s ratio of each element is set to $0.5$. The boundary and loading conditions of the body are shown in SI Appendix, Fig. S10. The top and bottom boundaries are free. The left boundary is fixed in such a way that the movements are not allowed along the horizontal direction (x-direction) but are allowed along the vertical direction (y-direction). To obtain a nonzero strain distribution of the body, external displacements along the x-direction are applied on the right boundary. The applied displacements are $1\%$ of the body length. Thus, an average normal strain along the x-direction (${\epsilon }_{xx}$) of $1\%$ can be generated. The strain distribution is calculated from the displacement distribution.

Encoded Domain Knowledge of Elasticity.

For a two-dimensional body, the relation between the strain and displacement with respect to the Cartesian axes is given by

$$\mathit{\epsilon }=\left\{\begin{array}{c}{\epsilon }_{xx}\\ {\epsilon }_{yy}\\ {\gamma }_{xy}\end{array}\right\}=\left\{\begin{array}{c}\frac{\partial u}{\partial x}\\ \frac{\partial v}{\partial y}\\ \frac{\partial u}{\partial y}+\frac{\partial v}{\partial x}\end{array}\right\},$$ [4]

where $\mathit{\epsilon }$ is the strain vector and $u$ and $v$ are the horizontal and vertical components of the displacement, respectively. The constitutive elasticity relation for a linear elastic isotropic material in plane stress is given by

$$\mathit{\sigma }=\left\{\begin{array}{c}{\sigma }_{xx}\\ {\sigma }_{yy}\\ {\tau }_{xy}\end{array}\right\}=\frac{E}{1-{\nu }^2}\left[\begin{array}{ccc}1& \nu & 0\\ \nu & 1& 0\\ 0& 0& \left(1-\nu \right)/2\end{array}\right]\left\{\begin{array}{c}{\epsilon }_{xx}\\ {\epsilon }_{yy}\\ {\gamma }_{xy}\end{array}\right\}\hspace{0.17em},$$ [5]

where $\mathit{\sigma }$ is the stress vector, $E$ is Young’s modulus, and $\nu$ is Poisson’s ratio. The equilibrium conditions are typically written in differential forms:

$$\frac{\partial {\sigma }_{xx}}{\partial x}+\hspace{0.17em}\frac{\partial {\tau }_{yx}}{\partial y}=0$$ [6]

and

$$\frac{\partial {\sigma }_{yy}}{\partial y}+\hspace{0.17em}\frac{\partial {\tau }_{xy}}{\partial x}=0.$$

These PDEs carry the derivatives of the stress field, which are the functions of the derivatives of the strain field and elasticity field. The derivatives of the strain field can be calculated from measured strains. However, measured strains naturally contain noise, and the calculation of the derivatives can amplify the noise significantly. The derivatives of the elasticity field cannot be calculated accurately when the derivatives of the strain field are inaccurate. To mitigate this potential problem, we rewrite the equilibrium conditions in finite difference forms:

$$\frac{{\sigma }_{xx}\left(x+\mathrm{\Delta }x,y\right)-{\sigma }_{xx}\left(x,y\right)}{\mathrm{\Delta }x}+\hspace{0.17em}\frac{{\tau }_{yx}\left(x,y+\mathrm{\Delta }y\right)-{\tau }_{yx}(x,y)}{\mathrm{\Delta }y}=0$$ [7]

$$\frac{{\sigma }_{yy}\left(x,y+\mathrm{\Delta }y\right)-{\sigma }_{yy}\left(x,y\right)}{\mathrm{\Delta }y}+\hspace{0.17em}\frac{{\tau }_{xy}\left(x+\mathrm{\Delta }\text{x},y\right)-{\tau }_{xy}\left(x,y\right)}{\mathrm{\Delta }x}=0.$$

From these equations, the equilibrium conditions for a small cube with sides of length $h$ can be expressed in Eq. 1.

Error Measurements in Predictions.

The error over the training epochs (Figs. 3C and 4C and SI Appendix, Figs. S1C and S2C) is quantified by the MAE, defined as

$$\text{MAE}=\frac{1}{n^2}{\displaystyle \sum }_i^n{\displaystyle \sum }_j^n{\text{error}}_{\text{absolute}}\left(i,j\right),$$ [8]

where $n$ is the dimension of the elasticity image in both x-direction and y-direction. ${\text{error}}_{\text{absolute}}$ is the absolute error of a prediction at each material point, defined as

$${\text{error}}_{\text{absolute}}\left(i,j\right)=\left|E_{\text{pred}}\left(i,j\right)-E_{\text{truth}}\left(i,j\right)\right|,$$ [9]

where $E_{\text{pred}}$ is the predicted elasticity and $E_{\text{truth}}$ is the ground-truth elasticity. The MAE can be used to compare the performances of different learning algorithms on the same model. However, the MAE may not be an ideal quantity to compare the accuracies between different models. A larger MAE can be expected when the mean elasticity of the model is larger. Therefore, we use the MRE to compare the accuracies between different models. The MRE is defined as

$$\text{MRE}=\frac{1}{n^2}{\displaystyle \sum }_i^n{\displaystyle \sum }_j^n\left|{\text{error}}_{\text{relative}}\left(i,j\right)\right|,$$ [10]

where ${\text{error}}_{\text{relative}}$ (unit: $\%$) is the relative error of a prediction at each material point, defined as

$${\text{error}}_{\text{relative}}\left(i,j\right)=100\times \left(E_{\text{pred}}\left(i,j\right)-E_{\text{truth}}\left(i,j\right)\right)/E_{\text{truth}}\left(i,j\right).$$ [11]

The relative error is used to generate the error maps (Figs. 3B and 4B and SI Appendix, Figs. S1B, S2B, and S6–S8).

ElastNet Training.

The DNN consists of $16$ fully connected hidden layers with $128$ neurons per layer. The rectified linear unit is adopted as the activation function. The input of the DNN is a vector of two variables ($x$, $y$) representing the position $p$ of a material point, and the output is the elasticity $E$ of the point. The stress $\sigma$ at a material point is calculated by the elastic constitutive relation based on the measured strain $\epsilon$ and predicted elasticity $E$. Full-batch learning is used when training the DNN. A predicted stress distribution $\mathit{\sigma }$ is generated when the entire data set is passed forward through the DNN. A convolutional layer consisting of $6$ filters of kernel size $3\times 3$ with stride $1$ is used to generate residual force maps. In these $6$ filters, $3$ of them are encoded for evaluating the equilibrium condition in the x-direction and the other $3$ are for the y-direction (Fig. 2A). The Adam optimizer (41) is adopted to minimize the loss function of the DNN. The loss function consists of two parts: one is from the residual forces and the other is from a physical constraint. The residual forces can be measured by the MAE. However, larger residual forces can be expected in areas with larger elasticity. Therefore, the accuracy (relative error) in areas with smaller elasticity will be compromised when using the MAE to measure the residual forces. To make the relative error maps more uniform, the normalized MAE is adopted to measure the residual forces when training the DNN to learn the hidden elasticity. The normalized MAE is defined as

$${\text{loss}}_{\text{force}}=\frac{1}{m^2}{\displaystyle \sum }_i^m{\displaystyle \sum }_j^m\frac{\left|e(i,j)\right|}{{\widehat{E}}_{\text{pred}}(i,j)},$$ [12]

where $m$ is the dimension of the residual force map in both x-direction and y-direction, $e$ is the residual force, and ${\widehat{E}}_{\text{pred}}$ is the sum of the predicted elasticity values in a cube containing $3\times 3$ material points. ${\widehat{E}}_{\text{pred}}$ is calculated as

$${\widehat{E}}_{\text{pred}}(i,j)={\displaystyle \sum }_{a=1}^3{\displaystyle \sum }_{b=1}^3{w}_E(a,b)E_{\text{pred}}(i+a-1,j+b-1),$$ [13]

where $w_E$ is a convolution kernel of ones. Two types of physical constraints are considered. One is based on the total applied force on the boundary, in which three force boundary conditions (BCs) can be added to the loss function. The penalty terms for these BCs are defined as

$${\text{loss}}_{\text{BC}1}={\displaystyle \sum }_i^m\left|\left({\displaystyle \sum }_j^m{\sigma }_{xx}(i,j)ht\right)-F\right|$$ [14]

$${\text{loss}}_{\text{BC}2}={\displaystyle \sum }_i^m\left|{\displaystyle \sum }_j^m{\tau }_{xy}\left(i,j\right)ht\right|$$

$${\text{loss}}_{\text{BC}3}={\displaystyle \sum }_j^m\left|{\displaystyle \sum }_i^m{\sigma }_{yy}\left(i,j\right)ht\right|,$$

where $F$ is the total applied force on the right boundary, which is the same as the total reaction force (along the x-direction) on the left boundary (SI Appendix, Fig. S10). These penalty terms can be applied to constrain the distribution of the internal stresses. The equilibrium conditions for these BCs are shown in the inner figure of SI Appendix, Fig. S1C. The other type of physical constraints is based on the mean elasticity. The penalty term for the mean elasticity is defined as

$${\text{loss}}_{\text{elast}}=\left|\frac{1}{n^2}{\displaystyle \sum }_i^n{\displaystyle \sum }_j^n{E}_{\text{pred}}\left(i,j\right)-\frac{1}{n^2}{\displaystyle \sum }_i^n{\displaystyle \sum }_j^n{E}_{\text{truth}}\left(i,j\right)\right|.$$ [15]

For the inclusion model (SI Appendix, Fig. S1), sinusoidal model (SI Appendix, Fig. S2), and the “Cal” model (Fig. 3), the DNN is trained for $\mathrm{200,000}$ epochs. For the Mona Lisa model (Fig. 4), due to the extremely complex hidden elasticity, the DNN is trained for $\mathrm{800,000}$ epochs. As the weights in the DNN are randomly initialized before training, the predicted elasticity distribution cannot be exactly the same when training with different initial weights. To better evaluate the performance of ElastNet, the predictions reported in this work are the average values after training the DNN $100$ times with different initial weights. The DNN is trained using TensorFlow (42) with a single GPU (NVIDIA Tesla V100 or Titan V).

Data Availability

All study data are included in the article and/or supporting information.