A Deep Learning Approach to Estimate Chemically-Treated Collagenous Tissue Nonlinear Anisotropic Stress-Strain Responses from Microscopy Images

Abstract

Biological collagenous tissues comprised of networks of collagen fibers are suitable for a broad spectrum of medical applications owing to their attractive mechanical properties. In this study, we developed a noninvasive approach to estimate collagenous tissue elastic properties directly from microscopy images using Machine Learning (ML) techniques. Glutaraldehyde-treated bovine pericardium (GLBP) tissue, widely used in the fabrication of bioprosthetic heart valves and vascular patches, was chosen to develop a representative application. A Deep Learning model was designed and trained to process second harmonic generation (SHG) images of collagen networks in GLBP tissue samples, and directly predict the tissue elastic mechanical properties. The trained model is capable of identifying the overall tissue stiffness with a classification accuracy of 84%, and predicting the nonlinear anisotropic stress-strain curves with average regression errors of 0.021 and 0.031. Thus, this study demonstrates the feasibility and great potential of using the Deep Learning approach for fast and noninvasive assessment of collagenous tissue elastic properties from microstructural images.

Graphical abstract

1. INTRODUCTION

Biological collagenous tissues are comprised of networks of collagen fibers embedded in a ground substance [1, 2], which provide pliability and strength important for many normal physiological functions. The attractive biological and mechanical properties [3] also make collagenous tissues, mostly derived from animals as xenografts, suitable for a broad spectrum of medical applications such as bioprosthetic heart valve (BHV) [4, 5], cardiovascular grafting/patch [6, 7], tendon [8] and hernia [9] repair. However, due to the heterogeneity and inherent variability of biological tissues, the mechanical properties of collagenous tissues obtained at different locations even within the same individual (regardless whether animal or human) may differ, and may impact tissue-derived device function.

Many studies [10–16] have shown that the microstructure of soft tissues, particularly the collagen fiber network structure, is the key determinant of the tissue elastic properties at the macroscopic level. Advanced microscopy imaging techniques, such as second harmonic generation (SHG) imaging, has enabled noninvasive visualization of soft tissue collagen networks at the microstructural level. The elastic properties of collagenous tissues are traditionally obtained through destructive mechanical testing of harvested tissue samples (Figure 1). Ideally, the nonlinear anisotropic elastic properties of collagenous tissues could be directly estimated from noninvasive images (e.g. SHG images) of the tissue microstructure, such that xenografts could be carefully selected based on their mechanical properties and optimal, more predictable, tissue-derived device function could be ensured.

Figure 1.

Two approaches to obtain the elastic properties of a tissue sample: 1) the traditional approach utilizing mechanical testing of a physical test sample and 2) noninvasive microscopy imaging coupled with a trained Deep Learning model.

Recently, Deep Learning [17–19], a branch of Machine Learning utilizing deep neural networks, has garnered enormous attention in the fields of artificial intelligence and image analysis. A special type of neural network, namely the convolutional neural network (CNN) [17, 20, 21], has become the state-of-the-art approach for computer vision and image analysis applications (e.g. face recognition), reaching, and even surpassing, human performance in some cases [22–25]. CNN provides an end-to-end solution from input image to output target value by automatically extracting image features, thus eliminating the need for hand-crafted image features.

In this study, we developed, to our best knowledge, the first Deep Learning approach to estimate the elastic properties of collagenous tissues from SHG images (Figure 1). Glutaraldehyde-treated bovine pericardium (GLBP) tissue, widely used in the fabrication of BHVs [5] and vascular patches, was chosen to develop a representative application for collagenous tissue elastic property estimation. A multi-layer CNN was designed and trained on a dataset of SHG images and corresponding mechanical testing results (i.e., equi-biaxial stress-strain curves). The trained CNN can automatically extract features from input SHG images of GLBP tissues and predict the nonlinear anisotropic elastic properties (Figure 1).

2. METHODS

2.1 Tissue preparation and mechanical testing

The GLBP tissue samples used in this study were collected and mechanically tested through previous work by our group aimed at evaluating transcatheter heart valve biomaterials [26]. The tissue preparation and mechanical testing protocols are well documented in the published works [27–29]. Briefly, testing samples were cut into a 20×20 mm² square, and four graphite markers delimiting a square approximately 2×2 mm² in size were glued to the central region of each sample for optical strain measurements. Samples were then mounted in a trampoline fashion to a planar biaxial tester in aqueous 0.9% NaCl solution at 37 °C. A stress-controlled test protocol [27] was utilized to obtain the biaxial stress-strain response curves of each testing sample. In this study, 48 GLBP tissue samples were tested in total.

2.2 Tissue imaging

Upon completion of biaxial mechanical testing, the tissue samples were imaged using the SHG technique at the unloaded state. We utilized a Zeiss 710 NLO inverted confocal microscope (Carl Zeiss Microscopy, LLC, Thornwood, NY, USA), equipped with a mode-locked Ti:Sapphire Chameleon Ultra laser (Coherent Inc., Santa Clara, CA), a non-descanned detector (NDD), and a Plan-Apochromat 40× oil immersion objective. The laser was set to 800 nm and emission was filtered from 380–430 nm. Samples were kept hydrated with saline solution during imaging to prevent drying artifacts and covered with #1.5 coverslips. Samples were imaged inside the area delimited by the graphite markers, and 2D image slices were collected in the thickness direction from the smooth side of each sample. A 2D slice has 512×512 pixels to 1024×1024 pixels, and for each sample the number of slices was varied to cover the thickness. In total, we obtained 3D SHG images (size from 512×512×N to 1024×1024×N) of 48 tissue samples from different animal subjects, and the corresponding mechanical testing data. Representative SHG images of a GLBP sample are shown in Figure 2, with a total of 18 slices (N=18) through the thickness. It can be seen that geometry variation (e.g. fiber waviness) in each imaging plane is much larger than that in the thickness direction as shown in Figure 2.

Figure 2.

Representative SHG image slices of a tissue sample. n denotes the index of each slice in the thickness direction.

2.3 PCA-based Parameterization of GLBP stress-strain curves

Two stress-strain curves were obtained from the equi-biaxial mechanical testing (section 2.1) of each tissue sample (Figure 3a&b): 1) strain E₁₁ and stress S₁₁ along the X₁-direction, and 2) strain E₂₂ and stress S₂₂ along the X₂-direction. As shown in Figure 3, the two curves of each sample are very different, indicating anisotropic mechanical behaviors. Each stress-strain curve was uniformly sampled along the stress axis within the range of 10 to 630 KPa. The cutoff of 630KPa was chosen because different ranges of external stresses were applied to the tissue samples and 630KPa was the minimum peak stress value [30]. For each tissue sample, the resampled strain values from the two curves were assembled as a vector of 126 numbers, Y. By using principle component analysis (PCA) [31–33], the vector Y of a tissue sample can be decomposed as

$$Y\cong Y_{\mathit{PCA}}=\overline{Y}+{\alpha }_1{V}_1+{\alpha }_2{V}_2+{\alpha }_3{V}_3$$ (1)

where $\overline{Y}$ is the population mean, {V_i} are the modes of variation, and {α_i} are the coefficients. Here, {α_i} can vary, while $\overline{Y}$ and {V_i} are the same for all tissue samples. From the PCA calculation [33], the first three modes of variation {V₁, V₂, V₃} with {α₁, α₂, α₃} can describe 99% of the total variation of the stress-strain curves, which means each stress-strain curve can be almost perfectly reconstructed by using Eq.(1) as shown in Figure 3b. Furthermore, the reconstruction error was measured by the mean absolute error (MAE), given by

$$\text{MAE}=\frac{1}{L_2-L_1+1}{\displaystyle {\sum }_{j=L_1}^{L_2}\left|Y_{\mathit{PCA}}(j)-Y(j)\right|}$$ (2)

where j is the index of a component in a vector; and if L₁= 1 L₂ = 63, MAE is the error of the reconstructed S₁₁~E₁₁ curve; and if L₁ = 64, L₂ = 126, MAE is the error of the reconstructed S₂₂~E₂₂ curve. Normalized mean absolute error (NMAE) was also calculated for reconstructed S₁₁~E₁₁ and S₂₂~E₂₂ curves by dividing MAE by the maximum value of E₁₁ and E₂₂, respectively.

Figure 3.

(a) The orientation definition of a tissue sample: X₁ direction and X₂ direction. (b) The open circles represent the stress-strain curves of a tissue sample from equi-biaxial mechanical testing experiments. The reconstructed stress-strain curves are shown by the red lines (S₁₁~E₁₁) and blue lines (S₂₂~E₂₂). (c)&(d) The stress-strain curves in the two directions of the 48 tissue samples color-coded by the corresponding α₁. The dashed lines are the mean curves, $\overline{Y}$.

As shown in Figure 3c&d, a material is softer, i.e. more compliant, than the mean material if α₁ < 0, and stiffer than the mean material if α₁ > 0. Thus the sign of α₁ can be used to describe the overall tissue stiffness.

2.4 Deep learning model

As show in Figure 4, we designed a deep convolutional neural network (CNN) as the deep learning model, consisting of 6 blocks in a pipeline. The 1^st block takes an input image of size 256×256×N pixels. The 6^th block can be configured either as a classifier of the overall tissue stiffness (sign of α₁), or a regressor to predict the PCA parameters {α₁, α₂, α₃,}, which can be used to reconstruct the stress-strain curves by Eq.(1). The CNN (Figure 4) learns the relationship between the tissue SHG images and elastic properties from the training dataset, and then can infer the elastic properties from a new tissue image.

Figure 4.

Architecture of the deep convolutional neural network used in this study.

Usually, convolutional neural networks (CNNs) consist of many layers that are sequentially connected, e.g., output from the first layer is the input to the second layer. A layer performs a specific operation, such as convolution, normalization, or max pooling, and it has parameters either prescribed or to be learned from data. For a detailed explanation of the layers and related theories, we refer the reader to the reference papers [17, 20, 34, 35] and the book [36]. The network structure should be designed for specific applications, e.g., choosing the types and sizes of layers and determining their combinations. For our application, the designed CNN consisting of 6 blocks in a pipeline, where each block has one or more layers. Given an input 3D image of 256×256×N pixels, the 1^st block with only one preprocessing layer, performs local contrast normalization and uniformly resamples the input 3D image into the first feature map of 256×256×3 pixels. The 1^st block does not have any trainable/free parameters. The 2^nd block contains a convolution layer with 64 filters (a.k.a. kernels) of 33×33×3 pixels, a batch-normalization layer, a ReLu (rectifier linear unit) layer, and a max pooling layer; and the output from the 2^nd block is a feature map of 32×32×64 pixels. The 3^rd to the 5^th blocks are very similar to the 2^nd block, which output feature maps of 16×16×64, 7×7×64, and 1×1×64 pixels respectively. All of the max-pooling layers use a 2×2 pooling window. The 1^st to 5^th blocks can be considered image-feature extractors which output a feature vector of 64 numbers. The 6^th block is used for classification with a softmax classifier, and regression with a linear model. The CNN was implemented by using MatConvnet [37], an open source MATLAB toolbox, and custom MATLAB functions; and it can process an input 3D image within 10 seconds on a PC with intel i7-4770 CPU and 32G RAM.

2.5 Learning of the deep convolutional neural network

The CNN (Figure 4) parameters were learned from the training data. To overcome the challenge of training the CNN with a small dataset [31] (i.e., 48 test samples, which is an acceptable sample size for material testing of biological tissues), the CNN was trained by combining: 1) unsupervised deep learning to determine the parameters in the 2^nd to 5^th blocks, 2) supervised learning to determine the parameters in the 6^th blocks, and 3) data augmentation to generate more training data.

2.5.1 Unsupervised Deep Learning from the 2^nd to 5^th blocks

To determine the filter parameters of a convolution layer, generally we could use encoder-decoder based unsupervised learning strategies [38–41]. The input feature map to the convolution layer can be divided into small patches, where each patch has the same size as a filter (all filters in the same layer have the same size). Each patch can be converted to a vector, X, and the vectorized patches can be stacked together as the columns of a data matrix X. The filters of the convolution layer can also be vectorized and stacked together as the columns of a filter matrix A. Let h(x) denote the ReLu function: h(x) = x if x > 0, and h(x) = 0 if x ≤ 0. The encoder performs convolution followed by ReLu to each patch, which outputs the code matrix h(AX) close to the optimal (unknown yet) code matrix Z. Given the optimal code matrix Z, the decoder tries to recover the input patches X by using a linear combination of the atoms/columns in a dictionary/matrix D, i.e, using DZ to approximate X. Then the goal is to find the optimal variables {A, D, Z} such that the encoding error and the decoding error are both minimized, which is to minimize the following objective function:

$$ℱ={\Vert h(\mathit{A}\prime \mathit{X})-\mathit{Z}\Vert }^2+{\Vert \mathit{X}-\mathit{DZ}\Vert }^2+g(\mathit{A},\mathit{D},\mathit{Z})$$ (3)

where g(A, D, Z) defines some constraints on the variables, and A′ denotes the matrix-transpose of A. The matrix norm ‖.‖ is the Frobenius norm. Obviously, by using different constraints, we can obtain different solutions of {A, D, Z}. We proposed an algorithm with three steps to directly obtain a solution under the low rank constraint [42]:

Step-1

Perform low rank approximation (LRA) [42] on the patches X, then a vectorized patch X can be approximated by

$$X\cong {\displaystyle {\sum }_{m=1}^M{z}_m{d}_m=\mathit{D}Z},$$ (4)

where D = [d₁, …, d_M], and the vector d_m has the same size as X, and $M\le \stackrel{\sim }{M}$ number of pixels in the patch X. d_m is the product of the m^th largest singular value, λ_m, and the corresponding left-singular vector obtained by LRA. D is the same for every single patch X. Also obtained by LRA, the code vector, Z = [z₁, … z_m]′, is a column vector of scalars, which is different for different patches. The percentage error of approximation for the patches X is given by

$$\mathit{Error}=\frac{{\displaystyle {\sum }_{n=M+1}^{\stackrel{\sim }{M}}{\lambda }_n^2}}{{\displaystyle {\sum }_{m=1}^{\stackrel{\sim }{M}}{\lambda }_m^2}}\times 100\%.$$ (5)

If $M=\stackrel{\sim }{M}$, then the error is zero. By controlling the number of retrained singular values and singular vectors, i.e., M, the approximation error and the computation cost (proportional to M) can be controlled. In this study, M is fixed to 32, and the error is less than 30%. The low rank approximation essentially obtains D and Z that minimize ‖X − DZ‖² under the low rank constraint. Since the singular vectors in D are orthogonal to each other, the code vector Z can be simply approximated by D′X, i.e., Z ≅ D′X, which is obtained by multiplying D′ to both sides of Eq.(4). After this step, the code matrix Z and dictionary D are determined.

Step-2

Define the filter matrix A by using the learned dictionary D, given by

$$\mathit{A}=[\mathit{D},-\mathit{D}]=[d_1,\dots ,d_M,-d_1,\dots ,-d_M].$$ (6)

Also, we define a new code vector $\stackrel{\sim }{Z}$ as

$$\stackrel{\sim }{Z}=[h(Z\prime ),h(-Z\prime )]\prime .$$ (7)

Then the objective function Eq.(3) is equivalent to

$$ℱ={\Vert h(\mathit{A}\prime \mathit{X})-\stackrel{\sim }{\mathit{Z}}\Vert }^2+{\Vert \mathit{X}-\mathit{D}\mathit{Z}\Vert }^2+g(\mathit{A},\mathit{D},\mathit{Z}),$$ (8)

where $\stackrel{\sim }{\mathit{Z}}$ is the stack of new code vectors, corresponding to Z, and $\stackrel{\sim }{\mathit{Z}}\cong h(\mathit{A}\prime \mathit{X})$ because Z ≅ D′X. Then, a vectorized patch X can be encoded as a vector $\stackrel{\sim }{Z}$ by the encoder h(A′X). For example, X ≅ 2d₁ − 3d₂, then the code vector is [2, 0] if A = [d₁, d₂], and the code vector is [2, 0, 0, 3] if A = [d₁, d₂ − d₁, −d₂] which clearly shows that the longer code vector preserves more information of X. The rationale of Eq.(6)&(7) is that the ReLu layer rejects any negative signal (i.e. code) output from the convolution layer, and therefore, nearly half of the signals will be lost in each block, harming the performance of the CNN. After this step, the filters of the convolution layer are determined.

Step-3

Perform feature map normalization. The output from the ReLu layer is a feature map serving as the input to the next layer. The size of the feature map is K₁ × K₂ × K₃ (i.e. height × width × channel). The values of the feature map at one spatial location can be assembled to a code vector Z of length K₃. By assembling all of the code vectors from the training dataset, a data matrix is obtained, and each row of this matrix is normalized by subtracting the mean and dividing by the standard deviation. The rows of the code matrix Z from a single input image will also be normalized in the same way by using the same values of mean and standard deviation. This normalization is essentially equivalent to batch-normalization [34] which has been shown to improve CNN accuracy. After this step, the parameters (i.e. mean and standard deviation values) of the normalization layer are determined.

2.5.2 Supervised learning in the 6^th block

The 6^th block can be configured either as a classifier or regressor. In the classification configuration, a softmax function is used to predict class membership based on the feature vector from the 5^th block. Since it is a binary (soft vs. stiff) classification task, the softmax function reduces to a logistic function, given by

$$y=\frac{1}{1+\text{exp}(-{\displaystyle {\sum }_{i=1}^{64}{w}_i{x}_i+b})}$$ (9)

where {w₁, …, w₆₄, b} are the unknown scalar parameters and [x₁, …, x₆₄] is the feature vector from the 5^th block. Usually, a discrimination threshold (e.g. 0.5) is specified for the binary classification. If y is greater than or equal to the threshold, then the input is classified as stiff; and if y is smaller than the threshold, then it is classified as soft. With the labeled training data (i.e., image data with known mechanical properties), the 65 parameters in of Eq.(9) can be determined through supervised learning using the cross-entropy loss function and the conjugate gradient optimization algorithm.

In the regression configuration, a multiple output linear regressor predicts the values of {α₁, α₂, α₃} in Eq.(1) based on the feature vector from the 5^th block, which is given by

$${\alpha }_i={\displaystyle {\sum }_{j=1}^{64}{w}_{ij}{x}_j+b_i,i=1,2,3}$$ (10)

where {w_ij, b_i, i = 1, 2, 3, j = 1, …, 64} are the unknown scalar parameters. With the labeled training data, the 195 parameters of this regressor can be learned by using the least squares regression algorithm. Once the parameters {α₁, α₂, α₃} are predicted by the regressor, the stress-strain curves can be reconstructed by using Eq.(1).

2.5.3 Data augmentation

Data augmentation methods are extensively used in Deep Learning applications [20, 43–45] to generate more training data. In this study, two data augmentation methods were used: image splitting and flipping (Figure 5). A 3D image of N slices can be split into patches using a sliding window with a stride of 128, and the size of each patch is 256×256×N. As a result of image splitting, 1678 patches were generated. Furthermore, each patch was flipped along the horizontal direction and/or vertical direction, which produced 6712 patches. The elastic properties corresponding to image patches from the same GLBP tissue sample, were assumed to be identical.

Figure 5.

An example of data augmentation to generate image patches.

2.6 A comparative study of network structures

Given the relatively large size of the CNN compared to the dataset, the natural question arises whether reducing the number of layers or filters will significantly impact the performance. Given the huge design space, it would be impractical to evaluate all possible simplifications of the CNN structure. In this study, we chose to investigate two simplified CNNs for comparison, named CNN-s1 and CNN-s2 respectively. CNN-s1: in Step-2 of unsupervised learning in section 4.5.1, the filter matrix A was simplified as A = D, which reduces the number of filters. CNN-s2: the ReLu and normalization layers were removed, and the filter matrix A was simplified as A = D. The structure of CNN-s2 is similar to that in [39].

3. RESULTS

3.1 Unsupervised deep learning

The learned filters of the CNN are visualized in Figure 6. The filters in the 2^nd block (Fig. 6a) are local image feature detectors, resembling the local fiber network structures. The filters in the other blocks (Fig. 6b–d) are more abstract, essentially representing various combinations of the local structures at different length scales and locations. For example, the filters in the 2^nd block have the same size of 33×33×3, and they will identify local structural patterns in the input image, and a feature map of size 32×32×64 will be produced to record the results of local structural pattern match. Then the filters in the 3^rd block, each of which has the size of 5×5×64, will operate on the feature map from the 2^nd block to produce a new feature map of size 16×16×64, which also records the results of pattern match. The filters in the 3^rd block resembles the patterns of the feature map from the 2^nd block, and therefore they evaluate the combinations of local structural patterns in the input image. By taking into account of the 2×2 max-pooling operation in the 2^nd block, the effective size (a.k.a receptive field size) of the filters in the 3^rd block is 69×69×3 measured in the space of the input image, much larger than the size of the filters in the 2^nd block.

Figure 6.

Examples of the learned filters. (a) The 32 filters in the convolution layer of the 2^nd block, the 32 opposites of these filters are not shown. The red box contains one filter (size is 33×33×3). (b) One of the filters in the convolution layer of the 3^rd block. (c) One of the filters in the convolution layer of the 4^th block. (d) One of the filters in the convolution layer of the 5^th block.

3.2 Classification

Classification performance was evaluated through ten-fold cross validation using the image patch data. In each round of cross validation, 90% of the image patches and corresponding overall stiffness values (i.e. sign of α₁) were randomly selected as the training data; and the remaining 10% of the data were used as the testing data to test whether the trained classifier can predict the sign of α₁, i.e., identify whether the tissue sample (corresponding to an image patch) is soft or stiff. The classification accuracy, defined as (TP+TN)/(TP+TN+FP+FN), the sensitivity, defined as TP/(TP+FN), and the specificity defined as TN/(TN+FP), were calculated to assess performance. Here, true positive (TP) is the number of stiff tissue patches correctly identified as stiff; false negative (FN) is the number of stiff tissue patches incorrectly identified as soft; true negative (TN) is the number of soft tissue patches correctly identified as soft; and false positive (FP) is the number of soft tissue patches incorrectly identified as stiff. In addition, AUC, defined as the area under a receiver operating characteristic (ROC) curve, was calculated as a measure of the overall classification performance. For comparison, a baseline softmax classifier using the skewness of image histogram [46] as the only feature, was also trained and tested. Since the two histograms of an image and its flipped version are the same, the flipped image patches were not used in the classification experiment. Two simplified versions of the CNN, CNN-s1 and CNN-s2 with less filters and less layers (details in Method section), were also tested.

ROC curves, as shown in Figure 7, were obtained by varying the discrimination threshold for each classifier. The performances of the proposed CNN, CNN-s1, CNN-s2, and the skewness-based softmax classifier using 0.5 as the discrimination threshold for classification, are reported in Table-1. The proposed CNN achieved the best performance, the skewness-based softmax classifier had the worst performance, and the two simplified CNNs had moderate performance.

Figure 7.

ROC curves of different classifiers. The ROC curve of an ideal classifier should be passing three points: the origin (0, 0), the top-left corner (0, 1) and the top-right corner (1, 1). The ROC curve of a random-guess classifier (e.g. tossing a coin to determine the stiffness) is just a straight line connecting the origin (0, 0) and the top-right corner (1, 1). The red curve, produced by the proposed CNN, is the closest to the ideal curve.

Table-1.

Classification Performance

Method	Accuracy	Sensitivity	Specificity	AUC
proposed CNN	84±2.5%	82±4.1%	86±3.6%	0.92
CNN-s1	78±2.8%	73±5.5%	80±3.7%	0.86
CNN-s2	75±3.5%	67±5.6%	80±4.9%	0.84
skewness based softmax classifier	71±3.2%	51±5.9%	84±3.9%	0.76

3.3 Regression

Regression performance was evaluated using a leave-one-out cross validation approach to test whether the trained regressor can predict the values of {α₁, α₂, α₃}, which were used to reconstruct the stress-strain curve of each tissue sample by Eq.(1). In each round of the cross validation, the image patches and the stress-strain curves from one of the 48 tissue samples were used as the testing data to evaluate the accuracy of the regressor, and the remaining data were used as the training data to determine the parameters of the regressor. The predicted {α₁, α₂, α₃} values for each of the image patches from the test tissue sample were averaged to obtain the final {α₁, α₂, α₃} predictions for the whole tissue sample.

From the cross validation, the MAE errors (Eq.(2)) in the predicted stress-strain curves were 0.021±0.015 and 0.031±0.029, and NMAE errors were 12.6±9.0% and 10.9±10%, compared to the actual S₁₁~E₁₁ and S₂₂~E₂₂ curves, respectively. Figure 8a shows an exemplary set of experimentally measured and predicted curves for one sample, and the error distribution across all of the samples is given in Figure 8b. The full set of predicted curves for all 48 samples are provided in the Supplementary Material.

Figure 8.

(a) Representative stress-strain curves predicted by the deep learning model shown as dashed lines, and the stress-strain curves obtained from mechanical testing shown as solid lines. S₁₁~E₁₁ curves are shown in red. S₂₂~E₂₂ curves are shown in blue. (b) The mean absolute error (MAE) distribution of all samples.

4. DISCUSSION

In this study, we developed a Deep Learning approach utilizing a deep CNN to estimate the elastic properties of collagenous tissues directly from noninvasive microscopy images. To our best knowledge, this is the first study in which Deep Learning techniques were used to derive nonlinear anisotropic elastic properties directly from tissue microscopy images. This work was motivated by the lengthy, complex, and destructive nature of traditional tissue mechanical testing. While it takes only about 10–30 minutes to obtain SHG images of a tissue sample, it takes much longer (hours) to prepare testing samples, set up testing and measurement instruments, and perform the actual mechanical test on each sample to obtain the stress-strain response curves. It took several months to obtain the data from the 48 samples used in this study. The success of this study holds promise for the use of Machine Learning techniques to noninvasively and efficiently estimate the mechanical properties of many biological materials as long as tissue mechanics are structurally determined and the structure can be noninvasively imaged (not necessarily using SHG).

Traditional machine learning methods [17] require hand-engineered features (i.e. features defined by human experts), which are difficult to obtain for this application. Rezakhaniha and co-workers [47] have defined intuitive texture features of tissue fibers, such as waviness, straightness, bundle size, etc., but this requires time-consuming manual annotation. Moreover, it is unclear whether these hand-engineered features could fully describe the fiber network structural information. As an end-to-end solution, CNN eliminates the need for hand-engineered features. One factor limiting the use of CNN and Deep Learning methods in biomechanics applications, is that they generally require a large amount of training data [48, 49], while the sample size for mechanical testing of biological tissues is typically small, on the order of 10 – 100 samples. However, in this study, it is shown that the deep CNN can also work well with a small dataset by combining supervised and unsupervised learning methods, and utilizing data augmentation methods. As more images and mechanical testing data are collected, the performance of the CNN can be further improved.

The CNN architecture used in this study, was specifically designed for this application. As explained in section 3.1, the 1^st to 5^th block of the CNN serve as automatic feature extractors that encode the input image into a feature vector of 64 numbers. The filters in the first convolution layer represent different local fiber network patterns, while the filters in the remaining convolution layers represent various combinations of these patterns at different locations and length scales. The 6^th block of the CNN relates the feature vector (i.e. code of the structure) to the elastic properties for classification and regression. Since the nonlinear relationship between tissue structure and elastic properties is described by the entire CNN (Figure 4), this approach is a structural approach: inferring mechanics from structure features extracted by CNN. Two simplified versions of the CNN were tested, i.e., CNN-s1 and CNN-s2 with less filters and less layers. The results show that simplifications to the CNN led to a significant decrease of accuracy, which may be the result of signal loss during signal propagation due to the fewer filters, and disruption of the encoding mechanism due to the fewer layers, respectively.

The CNN also demonstrated superiority over a simple image-feature based method to estimate the overall stiffness of collagen-based materials. Raub and co-workers [46] showed that the skewness of an image histogram was correlated to the collagen concentration and the Young’s modulus of collagen gels. Therefore, a softmax classifier was built by using the skewness as the only input feature in this study. As demonstrated in the results (Figure 7), the CNN outperformed the softmax classifier by a large margin; and even the two simplified versions of the CNN performed better than the softmax classifier, which underscores the superiority of CNNs for automatically extracting fiber network features.

More importantly, we demonstrated that the CNN can predict the PCA parameters of the stress-strain curves, such that the entire anisotropic stress-strain response of GLBP tissues can be estimated. For a nonlinear elastic response, it is well known that the Young’s modulus or stiffness cannot fully describe the tissue mechanical behavior, since the tangential value changes at different stress/strain levels along the nonlinear stress-strain curve. Thus, the PCA parameters offer a much more comprehensive look at the tissue elastic properties. Interestingly, we found that for this application, the overall “shape” of a stress-strain curve can be described with a single parameter, α₁ in Eq.(1). The novel PCA based approach to represent stress-strain curves developed in this study may facilitate more thorough analysis and comparison of tissue stress-strain responses over basic stiffness metrics.

The performance of the CNN can be further improved. There were some poor predictions which are mostly likely due to a common problem in machine learning: If the patterns of an input (i.e. tissue image in this application) are very unique compared to those of the training data, then the trained machine learning models will not recognize the input. As suggested by a recent work on the relationship between deep learning and training data [50], this problem can be solved by using more training data and fine tuning the neural network in our future work. In case of having extremely unique samples, a rejection option [51, 52] can be incorporated to the machine learning models, and the enhanced models can refuse to make a decision if the patterns of input are very different from those of the training data.

We note that there are many works in the literature on estimating the nonlinear mechanical properties of biological tissues by noninvasive or minimally invasive techniques. Most of these works are based on optimization strategies and constitutive models, for which the goal is to obtain the parameters of constitutive models, representing material mechanical properties. An energy-based method [53] was proposed to estimate myocardial stiffness from reduced 2D deformation data. An optimized virtual field method [54] was developed to identify myocardial stiffness from magnetic resonance elastography. Iterative finite element simulation and optimization methods [55, 56] were developed to determine human aorta material properties from strain measurements. Our group has also proposed a stress-matching based method [57] to estimate human aorta material properties. Recently, a machine learning based method [58], utilizing recurrent neural networks, was proposed for modelling the indentation force response of soft tissue, which used a robotic system to measure tissue forces and deformation. These methods can recover mechanical properties from multiple deformed states of the subjects by using noninvasive or minimally invasive measurement data from force sensors and motor controller [58], CT [57], MR [54] or ultrasound scanners [55, 56]. Temporal deformation measurement of the subjects under external loading is needed by these methods, which contains stress-strain information implicitly or explicitly. Unlike these prior efforts, the proposed approach in this study does not rely on deformation measurement data, and it utilized microcopy imaging to noninvasively reveal biological tissue structure at the unloaded state, which can be used to infer mechanical properties by the deep learning techniques. This approach suggests a new avenue for the fast and noninvasive assessment of collagenous tissue elastic properties from microstructural images, enabling many potential applications such as serving as a quality control tool to screen tissues [59] for the manufacturing of BHVs.

The method was only tested on one type of collagenous tissues (i.e. GLBP), which is the limitation of this study. Additional tissue types should be tested in future studies. However, based on the famous universal approximation theorem (UAT) and its variants [36, 60–63] in machine learning, it is theoretically feasible to apply this approach to other types of tissues. The original UAT [60] shows that any continuous function (i.e. input-output relationship) can be uniformly approximated by a neural network with sigmoid units. Recently, it is shown that a neural network with unbounded activation functions is still a universal approximator [61]. Thus, deep neural networks (e.g. CNN) have the capability to model complex relationship between structural image (input) and stress-strain curves (output) under the two conditions: 1) there is certain deterministic relationship between tissue structure and mechanics, and 2) tissue structure can be imaged. Although theoretically justified, the implementation of this approach for other tissues is non-trivial, which needs a substantial amount of work on tissue preparation, mechanical testing, imaging, and development of CNN models, and therefore it warrants a future study.

5. CONCLUSION

In conclusion, this study demonstrated the feasibility of using Deep Learning techniques for fast and noninvasive assessment of GLBP elastic properties from microcopy images. The main contributions of this study include: 1) the use of PCA to parameterize equi-biaxial stress-strain curves and quantify the overall stiffness, 2) the custom deep convolutional neural network design to automatically extract structural patterns of collagenous tissues, and perform classification to identify overall stiffness, as well as regression to predict PCA-parameters of nonlinear anisotropic stress-strain curves, and 3) the unsupervised deep learning method combined with supervised learning and data augmentation to overcome the challenge of small datasets for Deep Learning in the field of biomechanics. The developed approach was evaluated through cross validation, where an average classification accuracy of 84% and average regression errors of 0.021 and 0.031 were achieved. This study clearly demonstrates the great potential for Machine Learning techniques to estimate tissue mechanical properties solely through the use of noninvasive microcopy images.

Statement of Significance.

In this study, we developed, to our best knowledge, the first Deep Learning-based approach to estimate the elastic properties of collagenous tissues directly from noninvasive second harmonic generation images. The success of this study holds promise for the use of Machine Learning techniques to noninvasively and efficiently estimate the mechanical properties of many structure-based biological materials, and it also enables many potential applications such as serving as a quality control tool to select tissue for the manufacturing of medical devices (e.g. bioprosthetic heart valves).