Collagen Fiber Centerline Tracking in Fibrotic Tissue via Deep Neural Networks with Variational Autoencoder-based Synthetic Training Data Generation

Abstract

The role of fibrillar collagen in the tissue microenvironment is critical in disease contexts ranging from cancers to chronic inflammations, as evidenced by many studies. Quantifying fibrillar collagen organization has become a powerful approach for characterizing the topology of collagen fibers and studying the role of collagen fibers in disease progression. We present a deep learning-based pipeline to quantify collagen fibers’ topological properties in microscopy-based collagen images from pathological tissue samples. Our method leverages deep neural networks to extract collagen fiber centerlines and deep generative models to create synthetic training data, addressing the current shortage of large-scale annotations. As a part of this effort, we have created and annotated a collagen fiber centerline dataset, with the hope of facilitating further research in this field. Quantitative measurements such as fiber orientation, alignment, density, and length can be derived based on the centerline extraction results. Our pipeline comprises three stages. Initially, a variational autoencoder is trained to generate synthetic centerlines possessing controllable topological properties. Subsequently, a conditional generative adversarial network synthesizes realistic collagen fiber images from the synthetic centerlines, yielding a synthetic training set of image-centerline pairs. Finally, we train a collagen fiber centerline extraction network using both the original and synthetic data. Evaluation using collagen fiber images from pancreas, liver, and breast cancer samples collected via second-harmonic generation microscopy demonstrates our pipeline’s superiority over several popular fiber centerline extraction tools. Incorporating synthetic data into training further enhances the network’s generalizability. Our code is available at https://github.com/uw-loci/collagen-fiber-metrics.

Graphical Abstract

1. Introduction

Fibrosis is a pathological condition characterized by excessive generation and deposition of extracellular matrix (ECM) proteins, such as fibrillar collagen, in the tissue microenvironment. The amount, distribution, and structural organization of fibrillar collagen are important factors underlying the architecture and function of normal tissue as well as a wide array of pathological processes, including wound healing, aging, cancer, atherosclerosis, and diabetes (Ajeti et al., 2011; Kwon et al., 2008; Koehler et al., 2006; Nadiarnykh et al., 2010; Hanley et al., 2016). Cancer-associated fibrosis is critical to the tumor cellular microenvironment and has a significant impact on cancer development, including both inhibition and promotion of tumor progression (Fang et al., 2014; Chandler et al., 2019; Quante et al., 2011; Kalluri, 2016). For example, Tumor-Associated Collagen Signatures (TACS) are distinguished by patterns of collagen orientation and deposition during mammary tumor progression; these patterns are found to be associated with the prognosis of breast cancer (Provenzano et al., 2006; Conklin et al., 2011). Similar correlations are found in other cancer types such as ovarian, kidney, and pancreas (Koehler et al., 2006; Best et al., 2019; Drifka et al., 2016; Hanley et al., 2016; Ouellette et al., 2021). Most of the collagen biomarkers in these cancers were identified by assessing histological specimens, where measurements of the collagen topography such as fiber density, alignment, straightness, length, width, and association with tumors and glands are analyzed (Bredfeldt et al., 2014b,a; Xu et al., 2014a; Chen et al., 2019).

In recent years, second-harmonic generation (SHG) microscopy has emerged as a powerful label-free imaging technique and is regarded as one of the gold standards for fibrillar collagen imaging in research (Chen et al., 2012; Bancelin et al., 2014), in addition to staining techniques such as Picrosirius Red and Masson’s trichrome staining (Junqueira et al., 1979; Foot, 1933; Rich and Whittaker, 2017; Lattouf et al., 2014). Driven primarily by SHG imaging, quantitative image analysis algorithms have been developed to characterize collagen topology in research studies. Some widely used collagen fiber analysis software packages include FibrilTool (Boudaoud et al., 2014), CurveAlign/CT-FIRE (Bredfeldt et al., 2014b; Liu et al., 2017, 2020), OrientationJ (Rezakhaniha et al., 2012), CytoSpectre (Kartasalo et al., 2015), TWOMBLI (Wershof et al., 2021), SOAX (Xu et al., 2015), FiberFit (Morrill et al., 2016), and Ridge detection (Steger, 1998). The majority of existing methods rely on handcrafted algorithmic representations, including but not limited to curvelets, ridges, active contours, structure tensors, and nematic tensors, to trace collagen and calculate the topological properties of line-like structures. Measurements specific to individual fibers, such as width, length, and angle, along with global features like alignment and density, are subsequently calculated based on the results of the centerline extraction.

Despite the success of existing collagen image analysis techniques, several limitations hinder their effectiveness. Firstly, most of the current solutions are not automated, do not scale efficiently to large datasets, and so are not suitable for high-throughput studies. Users often have to manipulate multiple parameters to achieve satisfactory results under varying imaging conditions (e.g. different objectives). Furthermore, the selection of parameters is user-biased and handcrafted features like curvelets may inadequately represent collagen fibers’ topological features, resulting in sub-optimal fiber extraction.

Recent advances in deep learning infrastructure have enabled efficient training of deep neural networks on enormous datasets. When properly trained with large and diverse datasets, deep learning-based workflows can be more robust against image noise compared to conventional image analysis algorithms and generalize better. Therefore, deep learning has become the preferred solution for a variety of computational histopathology tasks, such as tumor detection (Campanella et al., 2019; Li et al., 2021; Hou et al., 2016), lesion segmentation (Sirinukunwattana et al., 2017), and cross-modality image synthesis (Keikhosravi et al., 2020; Bayramoglu et al., 2017). Although there have been deep learning-based methods for directly scoring fibrosis and desmoplasia in diseases including liver fibrosis, breast cancer, and scars (Sneider et al., 2022; Pham et al., 2021; Yu et al., 2018), these frameworks arguably lack interpretability due to the end-to-end prediction from the image domain to classification scores (e.g., diagnosis outcome). By contrast, a more explainable framework could instead target the extraction of individual fiber centerlines, from which morphological measurements such as collagen fiber density, alignment, length, and waviness can be obtained. These features can then be correlated with disease diagnosis and prognosis, providing insights regarding the structural characteristics involved in the biological process, such as tumor metastasis and drug delivery efficiency, and aiding the development of new therapies and drugs (Walsh et al., 2015; Tomko et al., 2018; Barcus et al., 2017; Valkenburg et al., 2018; Hosein et al., 2020; Plava et al., 2019). Compared to an end-to-end ”black box” model, these results can also be more easily translated into descriptive pathology reports interpretable by clinical pathologists. However, one major challenge in developing such a model is creating collagen annotations on a large scale, due to the intensive labor needed to annotate collagen fiber centerlines. A typical region of interest (ROI) can contain up to a few hundred collagen fibers and frequently present low-SNR regions and overlapping collagen fiber bundles, which further increase the complexity of annotating.

In this paper, we propose a deep learning-based pipeline for collagen fiber centerline tracking empowered by deep generative models that alleviates the problem of lacking large-scale annotations. Our pipeline consists of three stages, namely, generating synthetic centerline annotations (Stage I) and corresponding collagen image data (Stage II) before ultimately training the collagen fiber centerline extracting model (Stage III). Stage I consists of a pair of variational autoencoder (VAE) frameworks that can generate synthetic collagen fiber centerline masks from a limited number of manual fully-annotated collagen fiber images collected using SHG. The properties of generated collagen fiber centerlines, such as collagen fiber orientations, alignments, waviness, density, and lengths, can be controlled via an auxiliary branch. Stage II utilizes a conditional generative adversarial neural network (cGAN) to synthesize realistic collagen fiber images from the collagen fiber centerline masks and produce an enriched training dataset with synthetic image-mask pairs. Stage III then involves training a U-Net using the enriched dataset to extract collagen fiber centerlines from collagen image inputs.

Evaluations of the pipeline on an image dataset of 268 ROIs, including collagen fiber images from breast, pancreas, and kidney sections, show the deep learning-based collagen fiber extraction workflow produces collagen centerlines that better agree with human annotations compared to the commonly used ridge detection algorithm (Steger, 1998), biopolymer analysis toolbox SOAX (Xu et al., 2015), and the popular collagen fiber analysis toolbox CT-FIRE (Bredfeldt et al., 2014b; Liu et al., 2017, 2020). With the training data enriched by the VAE, the downstream centerline extraction network achieves better performance on the test images, with a soft Intersection over Union (IoU) of 0.7207. Additional evaluations on general VAE benchmark datasets show that our proposed VAE can consistently generate property-controlled images with a performance comparable to other recent models while having relatively simpler architecture and implementation. Our code is fully open source at https://github.com/uw-loci/collagen-fiber-metrics. In addition, we have made the human-annotated collagen image-mask data publicly available (Zenodo repository DOI https://doi.org/10.5281/zenodo.7243211), hoping to facilitate future collagen fiber analysis research.

2. Related works

The applications of collagen fiber analysis and quantification tools in pathological studies have been reviewed in the introduction. In this section, we focus on the existing VAEs frameworks and other related works using deep generative models to generate synthetic training data in biomedical imaging.

2.1. Data augmentation via image synthesis

Data augmentation aims to increase the size and diversity of training sets to increase model generalizability and reduce overfitting. Generating synthetic training data via deep generative models has shown to be an effective way to improve model performance in various medical image analysis tasks, such as classification and segmentation, where obtaining annotations and training data are challenging. (Moreno-Barea et al., 2020; Singh and Raza, 2021; Shorten and Khoshgoftaar, 2019; Chlap et al., 2021).

Commonly used deep generative models for synthetic training data generation are generative adversarial networks (GAN) (Goodfellow et al., 2014), variational autoencoders (VAE) (Kingma and Welling, 2013), and more recently, diffusion models such as Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020) and Denoising Diffusion Implicit Models (DDIM) (Song et al., 2020). GAN learns to transform random noise into realistic images via a generative network. The data distribution is implicitly learned using a surrogate loss from a discriminative network trained jointly with the generative network. Images generated by GANs are usually sharper and have more high-resolution details than those generated by VAE; however, controlling the generative process is inherently challenging, and the training process can often be unstable due to its adversarial nature. Nevertheless, many variants of GANs have demonstrated effectiveness in data augmentation for improving performances of medical image analysis tasks with applications ranging from CT scan (Frid-Adar et al., 2018), X-ray (Kong et al., 2022), and histopathology (Xue et al., 2021) images. Interested readers can refer to Shin et al. (2018); Singh and Raza (2021); Chlap et al. (2021) for thorough reviews on this topic.

On the other hand, VAE aims to learn latent representations explicitly from the data using autoencoders. One of the major advantages of VAEs over GANs is the easier control of the generation process through direct manipulation of latent variables. However, VAEs tend to generate images with extra blurriness. VAE-based data augmentations have also been used for generating synthetic training data in multiple biomedical imaging domains, including brain MRI (Chadebec et al., 2022; Chadebec and Allassonnière, 2021), eye-tracking (Elbattah et al., 2021), and spinal ultrasound (Pesteie et al., 2019). VAEs are also frequently used together with GANs in synthetic data generation. A VAE-GAN framework was proposed to synthesize blood vessel annotation masks and corresponding fundus images (Sengupta et al., 2020), and also for improving classifications of white blood cells (Liu et al., 2021). VAE and GAN-based models have been shown to outperform the generative baselines of Gaussian mixture models on an fMRI dataset Zhuang et al. (2019).

Recently, diffusion models (Ho et al., 2020; Song et al., 2020) have gained increasing attention in the computer vision community for synthesizing photorealistic images. The training process works by progressively disturbing data with noise, then learning to reverse the process for un-/conditional sample generation. Along with GAN and VAE, the diffusion models have also shown promising results in generations of realistic medical images including CT and X-rays (Ali et al., 2023; Chambon et al., 2022; Packhäuser et al., 2022; Pinaya et al., 2022).

2.2. Towards controllable VAE

Controlling the data generation process to generate data with desired properties using VAEs is an active research area (Bengio et al., 2013; Kingma and Welling, 2019; Liu et al., 2022). An important relevant line of research focuses on VAEs for unsupervised learning of disentangled representations (Higgins et al., 2017; Chen et al., 2018; Kim and Mnih, 2018; Mathieu et al., 2019). However, as indicated in Locatello et al. (2019a), learning disentangled representations in this way is fundamentally impossible without some form of inductive bias.

To achieve controllable data generation, semi-/supervised VAEs inject inductive bias to match the latent variables with the supervising labels (i.e., the underlying true generative factors) (Kingma et al., 2014; Maaløe et al., 2016; Paige et al., 2017; Ding et al., 2020). Other approaches combine adversarial learning with VAE to attain similar objectives (Makhzani et al., 2015; Mathieu et al., 2016; Creswell et al., 2017). On the other hand, instead of directly enforcing the latent variables to be equal to the label values, Joy et al. (2020) suggested capturing the salient significance of the labels via the supervision of latent variables. The majority of the works focus on categorical labels (Klys et al., 2018; Locatello et al., 2019b; Ilse et al., 2020). These approaches were extended to continuous-valued labels in work by Guo et al. (2020), which utilized reversible functions requiring iterative computations.

Similarly, we aim to control the generative process to generate data with desired properties by supervising a VAE using continuous-valued labels corresponding to the values of the fiber centerline properties (Sec. 3.4). We can then enrich our manually-annotated fiber centerline dataset containing synthetic centerlines with the specified properties. We achieve this by deploying an auxiliary VAE to supervise the main VAE that focuses on controlling the properties of the generated data. Unlike a similar formulation in Property Controllable VAE (PCVAE) (Guo et al., 2020), we assume the disentanglement among latent variables, which simplifies implementations by not requiring the additional disentanglement loss terms and iterative inference computations as in PCVAE, but instead having an additional regularization term that enforces the consistency between supervision and latent variables while keeping the total amount of information minimal. Nevertheless, we demonstrate similar property-controlled data generation performance (Sec. 4.2). The two VAEs cooperate as a duo to achieve the common goal of generating property-controlled synthetic data. We refer to this approach as DuoVAE.

3. Method

The proposed pipeline for collagen fiber centerline extraction using synthetic training data generated by DuoVAE and cGAN is illustrated in Fig. 1. In Stage I, the proposed DuoVAE is trained to generate collagen fiber centerline masks conditioned on the controlled centerline properties of interest (Fig. 1a). In Stage II, a generator of cGAN synthesizes realistic collagen fiber images from the generated centerline masks (Fig. 1b). Lastly, in Stage III, a U-Net is trained to perform centerline extraction, using a combination of the original and synthetic dataset (Fig. 1c–d).

Fig. 1:

The pipeline proposed for synthetic training data generation and collagen centerline extraction. First, (a) DuoVAE is trained to generate fiber centerline masks conditioned on the centerline properties $\mathbf{y}$ and latent variables $\mathbf{z}$, and (b) cGAN is trained to synthesize realistic collagen fiber images from the centerline masks. Then, we generate a synthetic training dataset by first generating synthetic collagen fiber centerline masks using the trained DuoVAE and generating realistic collagen fiber images based on the masks using the trained cGAN (c). Lastly, (d) the centerline extracting network (U-Net) is trained using a combination of the original and synthetic datasets.

3.1. Stage I: DuoVAE for property-controlled data generation

The objective of Stage I is to generate fiber centerline masks that exhibit the controlled centerline properties of interest. This can be achieved using our proposed DuoVAE comprised of the main and auxiliary VAEs. We first describe the model assumption and the optimization objective of a standard VAE and then introduce our DuoVAE with extensions for property-controlled data generation.

VAE and the Evidence Lower Bound (ELBO)

Given a dataset in which each sample $\mathbf{x}$ is an observed variable (e.g., an image), a VAE (Kingma and Welling, 2013) aims to estimate the distribution of the variable $p_{\theta }(\mathbf{x})=\int p_{\theta }(\mathbf{x},\mathbf{z})d\mathbf{z}=\int p_{\theta }(\mathbf{x}\mid \mathbf{z})p_{\theta }(\mathbf{z})d\mathbf{z}$ where $\mathbf{z}$ denotes the latent variables, $p_{\theta }(\mathbf{x}\mid \mathbf{z})$ is referred as the generative model (i.e., decoder) which reconstructs the input data $\mathbf{x}$ from the latent variables $\mathbf{z}$, and the prior $p_{\theta }(\mathbf{z})$ is assumed to be an isotropic multivariate Gaussian distribution. The log-likelihood of the data $\log p_{\theta }(\mathbf{x})$ can be expressed as (derivations in Appendix 6.1.1)

$$\begin{gathered} \text{log}{p}_{\theta }(x)=D_{KL}\left(q_{\varphi }(z\mid x)\parallel p_{\theta }(z\mid x)\right) \\ +\underset{ℒ(x)(\text{ELBO})}{\underbrace{{𝔼}_{q_{\varphi }(z\mid x)}[\text{log}{p}_{\theta }(x\mid z)]-D_{KL}\left(q_{\varphi }(z\mid x)\parallel p_{\theta }(z)\right)}}\ge ℒ(x), \end{gathered}$$ (1)

where $q_{\varphi }(\mathbf{z}\mid \mathbf{x})$ is the recognition model (i.e., encoder) introduced for approximating the intractable true posterior $p_{\theta }(\mathbf{z}\mid \mathbf{x})$. Due to the non-negativity of the Kullback-Leibler (KL) divergence $D_{KL}\left(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\parallel p_{\theta }(\mathbf{z}\mid \mathbf{x})\right)\ge 0$, estimating $p_{\theta }(\mathbf{x})$ can be achieved by maximizing the ELBO $ℒ(\mathbf{x})$ w.r.t. the learnable parameters $\theta$ and $\varphi$, which, in turn, simultaneously minimizes $D_{KL}\left(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\parallel p_{\theta }(\mathbf{z}\mid \mathbf{x})\right)$. In the setting of VAE, the variational distribution is parameterized by the encoder which itself is a neural network with variational parameters $\varphi$ (Kingma and Welling, 2019). The decoder is another neural network with variational parameters $\theta$. Instead of learning a set of variational parameters for each datapoint, the optimization procedure learns these two parametric functions with respect to the whole dataset using stochastic gradient descent (known as amortization in stochastic variational inference (Hoffman et al., 2013; Kingma and Welling, 2013; Gershman and Goodman, 2014)).

DuoVAE

Let us consider each sample in the dataset as $(\mathbf{x},\mathbf{y})$ where $\mathbf{x}$ is a collagen fiber centerline mask and $\mathbf{y}={\left\{y_k\in ℝ\right\}}_{k=1}^K$ is a set of $K$ continuous-valued properties calculated from $\mathbf{x}$ that are important characteristics of collagen fibers (introduced in Sec. 3.4). The encoder maps each centerline mask $\mathbf{x}$ to a set of latent variables $(\mathbf{z},\mathbf{w})$.

If the groups of latent variables $\mathbf{z}$ and $\mathbf{w}$ are mutually disentangled, we can assume that the information on $\mathbf{x}$ correlated to the property values $\mathbf{y}$ is encoded to $\mathbf{w}$ and everything else is encoded to $\mathbf{z}$; then, the decoder reconstructs the input $\mathbf{x}$ given $(\mathbf{z},\mathbf{w})$ where $\mathbf{w}$ is conditioned on the properties $\mathbf{y}$. The hope is by manipulating the property values $\mathbf{y}$, the latent variables $\mathbf{w}$ can be controlled, thus, the generated centerline mask $\mathbf{x}$ presents the desired properties $\mathbf{y}$ (Fig. 2a).

Fig. 2:

(a) The generative process modeled as $p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{w})p_{\theta }(\mathbf{z})p_{\theta }(\mathbf{w}\mid \mathbf{y})p_{\theta }(\mathbf{y})$ for generating a collagen fiber centerline mask $\mathbf{x}$ with desired properties of collagen fibers $\mathbf{y}$. (b) Main VAE - Left: inference model where we assumed the factorized posterior $q_{\varphi }(\mathbf{z}\mid \mathbf{x})q_{\varphi }(\mathbf{w}\mid \mathbf{x})$. The dashed arrow represents the auxiliary regressor $p_{\gamma }(\mathbf{y}\mid \mathbf{w})$, and the dashed line represents the KL-divergence between $q_{\varphi }(\mathbf{w}\mid \mathbf{x})$ and $p_{\theta }(\mathbf{w}\mid \mathbf{y})$. Right: generative model. (c) Auxiliary VAE with inference (left) and generative (right) models.

The generative process can then be expressed as $p_{\theta }(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{w})=p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{w})p_{\theta }(\mathbf{z})p_{\theta }(\mathbf{w}\mid \mathbf{y})p_{\theta }(\mathbf{y})$ with learnable variational parameters $\theta$, where $p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{w})$ is the decoder, $p_{\theta }(\mathbf{z})=𝒩(0,\mathbf{I})$ is a Gaussian prior, $p_{\theta }(\mathbf{w}\mid \mathbf{y})$ is interpreted as conditional prior or posterior, and $p_{\theta }(\mathbf{y})$ is the marginal likelihood of the property values $\mathbf{y}$. Furthermore, we assume the approximate posterior can be factorized as $q_{\varphi }(\mathbf{z},\mathbf{w}\mid \mathbf{x},\mathbf{y})=q_{\varphi }(\mathbf{z}\mid \mathbf{x},\mathbf{y})q_{\varphi }(\mathbf{w}\mid \mathbf{x},\mathbf{y})$, and since $\mathbf{y}$ can be derived from $\mathbf{x}$ plus some noise from annotations and does not impact the expectation terms, it is dropped when evaluating the first term of ${\overline{ℒ}}_1$ (Eq. 3) (Guo et al., 2020). Besides, such perfect disentanglement of $\mathbf{z}$ and $\mathbf{w}$ is not trivial to achieve in practice; however, we show empirically that when the regularization weight coefficients are carefully chosen, this formulation can achieve a degree of latent feature disentanglement and the quality and controllability of data generation is on a par with recently proposed models with more complex feature disentanglement mechanisms (Sec. 4.2 and Appendix 6.4.3).

For training the main VAE (Fig. 2b), the ELBO ${\overline{ℒ}}_1(\mathbf{x},\mathbf{y})$ can be written as (derivations in Appendix 6.1.2 and 6.2)

$$\begin{gathered} {\overline{ℒ}}_1(x,y)\propto {𝔼}_{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}[\text{log}{p}_{\theta }(x\mid z,w)]-{\beta }_z{D}_{KL}\left(q_{\varphi }(z\mid x)\parallel p_{\theta }(z)\right) \\ -D_{KL}\left(q_{\varphi }(w\mid x)\parallel p_{\theta }(w\mid y)\right)+\text{log}{p}_{\theta }(y), \end{gathered}$$ (2)

where we introduced the regularization coefficient ${\beta }_z$ similar to $\beta$-VAE (Higgins et al., 2017) to control the capacity of the latent information in $\mathbf{z}$. Maximizing ${\overline{ℒ}}_1(\mathbf{x},\mathbf{y})$ requires the log-likelihood of the properties $\log p_{\theta }(\mathbf{y})$ and the intractable posterior $p_{\theta }(\mathbf{w}\mid \mathbf{y})$ in the KL-divergence term $D_{KL}\left(q_{\varphi }(\mathbf{w}\mid \mathbf{x})\parallel p_{\theta }(\mathbf{w}\mid \mathbf{y})\right)$ to be known.

To evaluate $D_{KL}\left(q_{\varphi }(\mathbf{w}\mid \mathbf{x})\parallel p_{\theta }(\mathbf{w}\mid \mathbf{y})\right)$, given the properties $\mathbf{y}$ are also observed variables and assuming $p_{\theta }(\mathbf{y})=\int p_{\theta }(\mathbf{y}\mid \mathbf{w})p_{\theta }(\mathbf{w})d\mathbf{w}$, we introduce an auxiliary VAE (Fig. 2c) comprised of the encoder $q_{\delta }(\mathbf{w}\mid \mathbf{y})$ and decoder $p_{\gamma }(\mathbf{y}\mid \mathbf{w})$ (where $\delta$ and $\gamma$ are also learnable parameters) to estimate $\log p_{\theta }(\mathbf{y})$, which simultaneously approximates the intractable posterior as $q_{\delta }(\mathbf{w}\mid \mathbf{y})\approx p_{\theta }(\mathbf{w}\mid \mathbf{y})$. Then, the objective of the auxiliary VAE is to maximize the ELBO ${ℒ}_2(\mathbf{y})$ of the log-likelihood $\log p_{\theta }(\mathbf{y})$, written as (derivations in Appendix 6.1.3)

$${ℒ}_2(y)\propto {𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}{p}_{\gamma }(y\mid w)\right]-{\beta }_w{D}_{KL}(q_{\delta }(w\mid y)\parallel p_{\theta }(w)),$$ (3)

where $p_{\theta }(\mathbf{w})$ is assumed to be a Gaussian prior, and ${\beta }_w$ is a regularization coefficient.

Furthermore, to encourage encoding of the information related to $\mathbf{y}$ from $\mathbf{x}$ into the latent variables $\mathbf{w}$, we introduce an auxiliary objective ${𝔼}_{q_{\varphi }(\mathbf{w}\mid \mathbf{x})}\left[\log p_{\gamma }(\mathbf{y}\mid \mathbf{w})\right]$ to ${\overline{ℒ}}_1(\mathbf{x},\mathbf{y})$, similar to Zhao et al. (2019); Ilse et al. (2020); Joy et al. (2020):

$${ℒ}_1(x,y)={\overline{ℒ}}_1(x,y)+\alpha {𝔼}_{q_{\varphi }(w\mid x)}\left[\text{log}{p}_{\gamma }(y\mid w)\right],$$ (4)

where $p_{\gamma }(\mathbf{y}\mid \mathbf{w})$ is the decoder of the auxiliary VAE, and $\alpha$ is a weight coefficient.

For training, we alternate between maximizing ${ℒ}_1(\mathbf{x},\mathbf{y})$ over the parameter set $\{\theta ,\varphi ,\gamma \}$ using the main VAE, and ${ℒ}_2(\mathbf{y})$ over the parameter set $\{\delta ,\gamma \}$ using the auxiliary VAE (Fig. 1a). When training the main VAE, the encoder of the auxiliary VAE $q_{\delta }(\mathbf{w}\mid \mathbf{y})$ is used as the approximate posterior in substitution for the posterior $p_{\theta }(\mathbf{w}\mid \mathbf{y})$ in ${\overline{ℒ}}_1(\mathbf{x},\mathbf{y})$.

3.2. Stage II: cGAN for collagen image generation from centerlines

The objective of Stage II is to synthesize a realistic collagen image $\overline{\mathbf{x}}$ from a centerline mask $\mathbf{x}$ generated by the VAE in Stage I. This can be achieved using the conditional GAN (cGAN) framework proposed by (Isola et al., 2017) which employs U-Net (Ronneberger et al., 2015) for the generator $f_G$ to generate an output image $\overline{\mathbf{x}}$ (i.e., $f_G:\{\mathbf{x},\epsilon \}\to \overline{\mathbf{x}}$) from an input image $\mathbf{x}$ and a random noise vector $\epsilon$ and PatchGAN for the discriminator $f_D$ (Fig. 1b).

PatchGAN encourages to model high-frequency structures and has a convolutional architecture which classifies each of the $N\times N$ local patches in an image as real or fake (Isola et al., 2017). Furthermore, we regard the generator $f_G$ as a deterministic mapping following Isola et al. (2017), providing the random noise $\epsilon$ only in the form of dropouts during training. The objective of cGAN is expressed as

$$\begin{gathered} {ℒ}_{cGAN}(f_G,f_D)={𝔼}_{x,\overline{x}}[\text{log}{f}_D(x,\overline{x})] \\ +{𝔼}_x[\text{log}(1-f_D(x,f_G(x,\epsilon )))], \end{gathered}$$ (5)

where the optimization process is executed in an adversarial manner, i.e., $\arg {\min }_{f_G}{\max }_{f_D}{ℒ}_{cGAN}\left(f_G,f_D\right)$. Moreover, we include pixel-wise L1 loss term ${ℒ}_{L1}\left(f_G\right)={𝔼}_{\mathbf{x},\overline{\mathbf{x}}}\left[{∥\overline{\mathbf{x}}-f_G(\mathbf{x},\epsilon )∥}_1\right]$ to encourage low-frequency reconstruction correctness. The final objective is expressed as

$$f_G{}^{*}=\text{arg}\underset{f_G}{\text{min}}\underset{f_D}{\text{max}}{ℒ}_{cGAN}(f_G,f_D)+\lambda {ℒ}_{L1}(f_G),$$ (6)

where $\lambda$ is a weight coefficient. Note that this stage only relies on the original collagen fiber image-mask pairs for training. Experimentally, we find cGAN generalizes to synthetic centerline masks produced by the VAE, possibly due to the fact that the cGAN with the PatchGAN setup learns low-level style information that is mostly localized and “decorates” the centerlines with these styles to perceptually match realistic collagen images (Ledig et al., 2017; Isola et al., 2017; Jing et al., 2019).

3.3. Stage III: Centerline extraction network

The goal of the centerline extraction network is to extract collagen fiber centerlines from a collagen image acquired by SHG. This can be achieved by training a neural network $f_{\psi }$ parameterized by a parameter set $\psi$ with input being the collagen image and output being the corresponding centerline mask (Fig. 1c). The training process then minimizes the empirical risk on the training set, expressed as

$$f_{\psi }{}^{\text{*}}=\text{arg}\underset{\psi }{\text{min}}{𝔼}_{\overline{x},x}\left[{ℒ}_{recon}\left(x,f_{\psi }(\overline{x})\right)\right],$$ (7)

where we employed a binary cross-entropy loss for ${ℒ}_{\mathit{recon}}$ and U-Net (Ronneberger et al., 2015) for $f_{\psi }$ with the number of input and output channels both equal to 1.

All of the optimizations in Stage I, II, and III were carried out using stochastic gradient descent with the Adam solver (Kingma and Ba, 2014).

3.4. Collagen fiber centerline property computing

The collagen fiber centerline properties of interest include the following 6 properties: 1) Fiber circular mean: average orientations of the fibers; 2) Fiber circular variance: alignments of the fibers (i.e., variation in the orientations); 3) Fiber density: compactness of the fibers; 4) Fiber waviness: average curviness of the fibers; 5) Fiber length mean: average length of the fibers; and 6) Fiber length variance: variation in the lengths.

These features are suggested in CT-FIRE (Bredfeldt et al., 2014b; Liu et al., 2017, 2020) and have been used in many pathological studies for characterizing collagen fibers. To compute their values, we first extract the individual fiber instances from the fiber centerline masks.

Extracting individual fiber instances

The output of the centerline extraction network $f_{\psi }$ in Stage III (i.e., U-Net) is a binary image where the locations of pixel value 1 can be considered as coordinates of collagen fiber centerlines without distinguishing between different fiber instances. To produce individual collagen fiber extraction results, the outputs of U-Net need to be further processed. The goal is to compute lists of coordinates where each list contains a sequence of coordinates that define a single fiber centerline.

The outputs of U-Net are first skeletonized using a simple thinning algorithm (Zhang and Suen, 1984) and joints are identified by a template matching filter that enumerates all possible conformations of joints in a $3\times 3$ kernel. Joints are locations where multiple collagen fibers intersect. To obtain individual fibers, segments of centerlines belonging to the same fiber at a joint need to be linked. The linking of centerline segments at joints is done using a greedy algorithm. A segment is randomly selected and linked to another segment with the most compatible opposite incident angle (i.e., closest to 180°). The opposite angle of the linked segment has to be in the range of 180°±30°). The process is repeated until all link-able segments are linked. This procedure is illustrated in Fig. 3.

Fig. 3:

The top row illustrates the algorithm that generates fiber instances from a centerline extraction output. The algorithm first detects joints and generates fiber instances by linking fiber segments at joints. Fiber segments with matching incident angles are connected. Collagen fiber properties can be computed based on individual fiber instance extraction results. The bottom row shows a representative processing result of fiber instances extraction.

Centerline properties

Fiber properties including fiber circular mean, fiber circular variance, fiber density, fiber waviness, fiber length mean, and fiber length variance can be computed by aggregating the respective properties computed for each fiber. The methods used to compute the properties primarily follow the CT-FIRE toolbox (Bredfeldt et al., 2014b; Liu et al., 2017, 2020). Fiber circular mean is defined as the weighted averaged angle of collagen fiber centerlines, computed as follows: (1) divide each fiber centerline into several equal-length sub-segments, and compute the averaged angle of the sub-segments; (2) average the angles of all centerlines in an image with each weight equaling the number of sub-segments in the centerline. Fiber circular variance is defined as the circular variance of all the sub-segments. Fiber density is the number of positive pixels (pixels on centerlines) in the image (all with a size of 256×256). Fiber waviness is the sum of absolute differences between the angles of adjacent sub-segments in the centerline of each fiber. Fiber length mean is the average length of all centerlines, and fiber length variance is the standard deviation of the fiber lengths. The features are computed for all images in the dataset and the values of each feature are standardized to have a zero mean and a unit variance.

4. Experiments and results

In this section, we first describe the specimens and images of the collagen fiber dataset, the pre-processing of the collagen dataset, as well as two additional VAE benchmark datasets used for evaluating DuoVAE (Sec. 4.1). We then present the evaluation results of (1) using DuoVAE for property-controlled data generation on both the collagen fiber dataset and the two VAE benchmark datasets (Sec. 4.2), and (2) the collagen fiber centerline extracting network trained on the VAE-enriched collagen fiber dataset (Sec. 4.3). Lastly, we present ablation experiments results to show the impacts of different weights ${\beta }_z$ and ${\beta }_w$ in the loss functions in Eq. (2) and (3) on latent feature disentanglement and data generation controllability as well as the performance when different amounts of synthetic training data are used for training the fiber extraction network (Appendix 6.4.3).

4.1. Datasets and pre-processing

4.1.1. Collagen fiber image dataset

Specimen and images

The collagen fiber image dataset consists of SHG images collected on tissue sections of three different organs - breast, pancreas, and kidney. The images were collected in several previously published cancer studies from three different imaging platforms in the Eliceiri lab. Details regarding the sample preparation, case demographics, and imaging systems can be found in Conklin et al. (2011); Bredfeldt et al. (2014b); Drifka et al. (2016); Best et al. (2019); Li et al. (2022). The 268 ROIs were randomly selected from the whole image set for annotating, excluding empty backgrounds and low-quality images that contain over-saturated signals caused by inappropriate settings in imaging. The image set contains 116 ROIs from breast tissues (ductal carcinoma), 89 ROIs from pancreas tissues (ductal adenocarcinoma and chronic pancreatitis), and 63 ROIs from kidney tissues (renal cell carcinoma). These images were split into a training set with 227 ROIs and a testing set with 41 ROIs. All ROIs have a size of 512 × 512 and are non-overlapping.

Image augmentation

Additional image augmentation workflows are implemented for the training phases of Stage I and Stage II (Sec. 3.1–3.2), and Stage III (Sec. 3.3).

Stage I (DuoVAE).

DuoVAE takes as inputs annotated fiber centerlines (in the form of image masks) with their corresponding 6 property values (orientation, alignment, density, waviness, average length, and length variation) computed using the method in Sec. 3.4. Given a set of manually annotated centerline masks of size 512 × 512, each mask is subdivided into 5 base crops of size 256 × 256 at different locations as illustrated in Fig. 4a and further augmented 8 times with 4 rotations {0°, 90°, 180°, 270°} and 2 horizontal flips (totalling 40 crops per image). Furthermore, $N$ number of crops are extracted at random locations with random rotations and horizontal flips within the image boundary as illustrated in Fig. 4b. This yields $40+N$ number of crops per annotated ROIs. Lastly, empty crops with no centerline pixels inside are removed, and the centerline properties are computed using the method in Sec. 3.4 and standardized for each property.

Fig 4:

Image augmentations during training of DuoVAE, where (a) each of the manually annotated 512 × 512 centerline masks is subdivided into 5 base crops of size 256 × 256, and then (b) additional N random crops of size 256 × 256 are extracted.

From 268 manually annotated ROIs, the numbers of base crops are 1,188 for the training set and 199 for the test set, after removing the empty crops. With rotation and flipping of the base crops, together with $N=100$ random crops on each ROI, we produce a training set with 36k crops of size 256 × 256. Note that additional augmentation is not performed on the test set (the test set consists of 199 base crops of size 256×256). The augmentation process is performed offline prior to the training of DuoVAE.

Stage II (cGAN).

Training Stage II requires pairs of fiber centerline masks and corresponding collagen images. Since the centerline properties do not need to be precomputed from the masks, we extract a crop on the fly during training using the same random augmentation method (cropping, rotations, and flipping) used for Stage I (Fig. 4b) for the mask-image ROI pairs.

Stage III (U-Net).

Training Stage III requires pairs of collagen images and corresponding fiber centerline masks. The 1,188 base crops obtained from the training ROIs are used to train the centerline extraction network. Another 4,752 synthetic training samples are generated by traversing the property labels of the 1,188 base crops using DuoVAE and mapping the centerlines to realistic collagen images using cGAN, resulting in the total number of 5,940 enriched training samples (i.e., 5× more training samples compared to the original 1,188 training samples). Additional ablation studies regarding the impacts of different augmentation ratios (2×, 5×, and 10× compared to the original data size) can be found in Appendix 6.4.3.

To generate the synthetic training data, each of the 1,188 base crops is first encoded into latent variables $(\mathbf{z},\mathbf{w})$ using the encoder of the main VAE. Then, the traversing for each of the 6 ground-truth properties $y_i\in \mathbf{y}$ is done one at a time while the rest of the untraversed ${\mathbf{y}}_{j\ne i}$ and the encoded $\mathbf{z}$ are held constant. The encoder of the auxiliary VAE then encodes a new set of latent variables ${\mathbf{w}}^{\prime }$ from different values of $\mathbf{y}$, and the decoder of the main VAE generates a synthetic crop from $\left(\mathbf{z},{\mathbf{w}}^{\prime }\right)$. Similarly, while holding the encoded $\mathbf{w}$ constant, values of $\mathbf{z}$ are randomly sampled from $𝒩(0,\mathbf{I})$, denoting as ${\mathbf{z}}^{\prime }$, and a synthetic crop is generated from the new set of latent variables $\left({\mathbf{z}}^{\prime },\mathbf{w}\right)$. In our example, we traversed each of the 6 properties $\mathbf{y}$ by 5 different values (i.e., 1, 188 × 6 × 5 = 35,640 crops) and also sampled $\mathbf{z}5$ times (i.e., 1, 188 × 5 = 5,940 crops), making the total of 41, 480 synthetic crops. Then, we randomly sampled 4, 752 crops to obtain the 5,940 enriched training samples (i.e., augmentation ratio of 5×).

4.1.2. VAE benchmark dataset

In addition to our collagen dataset, we include two additional VAE benchmark datasets to validate further that our DuoVAE shows consistent performance in more general settings. The dSprites dataset (Matthey et al., 2017) and 3dshapes dataset (Burgess and Kim, 2018) consist of 737k images of 2D shape and 480k images of 3D shapes, respectively, procedurally generated from 6 ground truth independent latent properties. For evaluations, we experimented with 3 properties for each dataset: (scale, $x$ position, $y$ position) for dSprites and (scale, wall color, floor color) for 3dshapes. For our collagen dataset, we experimented with all 6 properties: orientation, alignment, density, waviness, average length, and length variation.

4.2. Evaluation of property-controlled data generation

We first visualize the property-controlled data generation of DuoVAE qualitatively and further present quantitative evaluation results that demonstrate the supervised latent variables have successfully captured the information of the corresponding properties.

4.2.1. Qualitative evaluation

We show the images generated when traversing each of the ground truth property values $y_i\in \mathbf{y}$ on the three datasets. Specifically, the encoder of the main VAE encodes an image into latent variables $(\mathbf{z},\mathbf{w})$; then, each of the ground truth properties $\mathbf{y}$ is traversed from its minimum to maximum values while holding $\mathbf{z}$ constant. The encoder of the auxiliary VAE encodes a new set of latent variables ${\mathbf{w}}^{\prime }$ from different values of $\mathbf{y}$. The decoder of the main VAE then generates new images from $\left(\mathbf{z},{\mathbf{w}}^{\prime }\right)$. For the collagen fiber dataset, we also visualize the collagen images generated by the cGAN that maps the fiber centerlines generated by DuoVAE into realistic collagen fiber images.

Fig. 5 shows representative outputs of the property-controlled data generation of DuoVAE on the collagen fiber dataset. The outputs are grouped into 6 panels according to the fiber properties: (a) orientation (from left-oriented to right-oriented), (b) alignment (from well-aligned to randomly organized), (c) density (from sparse to dense), (d) waviness (from straight to wavy), (e) average length (from short to long), and (f) length variation (from uniform lengths to random lengths). The first rows in each panel are the fiber centerlines generated by DuoVAE, and the second rows are the corresponding collage image generated by the cGAN. The generated centerlines exhibit the different values of fiber properties specified in the generating process.

Fig. 5:

Representative property traversing of collagen fiber centerline generation results of the VAE (top row of each panel) and the corresponding collagen fiber images generated by the cGAN (bottom row of each panel). (a): Traversing of the variable that encodes average fiber angles. (b): Traversing of the variable that encodes circular variance of fiber angles. (c): Traversing the variable that encodes fiber density. (d): Traversing of the variable that encodes average fiber waviness. (e): Traversing of the variable that encodes average fiber lengths. (f): Traversing of the variable that encodes the variation of fiber lengths.

Similarly, each row in Fig. 6a and Fig. 6b exhibits the controlled properties of interest where $\left(y_1,y_2,y_3\right)$ are (scale, $x$ position, $y$ position) for the dSprites dataset and (scale, wall color, floor color) for the 3dshapes dataset.

Fig. 6:

Representative property traversing results. (a) Images generated by traversing properties of interest on the dSprites dataset. $y_1$: scale of the object (from small to large). $y_2:x$ position (horizontal) of the object (from left to right). $y_3:y$ position (vertical) of the object (from top to bottom). (b) Images generated by traversing properties of interest on the 3dshapes dataset. $y_1$: scale of the object (from small to large). $y_2$: color of the wall in the background (rainbow color wheel from red to violet). $y_3$: color of the floor (rainbow color wheel from red to violet).

4.2.2. Quantitative evaluation

We present quantitative evaluation results of DuoVAE using the dSprites and 3dshapes datasets. Normalized mutual information is computed between each of the properties - (scale, $x$ position, $y$ position) for dSprites and (scale, wall color, floor color) for 3dshapes - and the corresponding supervised latent variables $\mathbf{w}$, as well as the average mutual information between each property and the rest of the latent variables $\mathbf{z}$. This metric was suggested in recent studies (Locatello et al., 2019b; Guo et al., 2020) for the purpose of evaluating the ability of controlling the properties of generated data in VAEs (Klys et al., 2018; Locatello et al., 2019b). We show that DuoVAE achieves comparable performance to other recent VAE-based models including Property Controllable VAE (PCVAE) (Guo et al., 2020), Conditional Subspace VAE (CSVAE) (Klys et al., 2018), and semi-VAE (Locatello et al., 2019b). Among these, PCVAE learns the correlation between each value of latent variables and properties by utilizing invertible constraints, which require iterative computations, and has shown to outperform the other two models both quantitatively and qualitatively.

Fig. 7 shows the normalized mutual information (MI) heatmaps by DuoVAE and PCVAE for the dSprites dataset (The details of the network backbones used for both models are provided in Appendix 6.3). Each entry denotes the MI between a property and a set of supervised latent variables $\mathbf{w}$ except the last row ${\mathbf{z}}_{\mathit{avg}}$ which denotes the averaged MI between a property and all the other latent variables $\mathbf{z}$ (other than $\mathbf{w}$). In the ideal case, the MI in the diagonal of the heatmaps should be 1, and off-diagonal values as well as ${\mathbf{z}}_{\mathit{avg}}$ should be zero (indicating perfectly encoded correlations where each property is completely inferred by one supervised latent variable $w_i$). The results show that the MI score is high between the supervised latent variables $\left(w_1,w_2,w_3\right)$ and their associated properties (scale, $x$ position, $y$ position), whereas the score is low for the rest of the unrelated latent variables including $\mathbf{z}$.

Fig. 7:

The heatmaps of the normalized mutual information between each of the properties and latent variables for dSprites dataset. The right two heatmaps are our evaluation results, and the left three are the reported results from the original PCVAE paper.

Moreover, DuoVAE attains higher MI scores compared to PCVAE, which suggests that the performance of DuoVAE is comparable to other recent models, though having relatively simpler architecture and implementation. We further visualize the images generated by traversing each latent variable. Fig. 8 shows that each of the supervised latent variables $\mathbf{w}$ (top 3 rows) captures the information of (scale, $x$ position, $y$ position), respectively. On the other hand, the rest of the latent variables $\mathbf{z}$ (bottom 4 rows) appear to have captured the rest of the information (e.g., rotation and shape) in an entangled way. The visualization for PCVAE can be found in Appendix 6.4.1, and the MI score and the representative visualization of the latent variable traversing on 3dshapes for DuoVAE and PCVAE can be found in Appendix 6.4.2.

Fig. 8:

Representative latent variable traversing by DuoVAE. Each of the supervised latent variables $\mathbf{w}$ (top 3 rows) captures the information of (scale, $x$ position, $y$ position), respectively, whereas the rest of the latent variables $\mathbf{z}$ (bottom 4 rows) appear to have captured the rest of the information (e.g., rotation and shape) in an entangled way.

4.3. Evaluation of collagen fiber centerline extraction

In this subsection, we present the evaluation results of the collagen fiber centerline extraction networks. We describe the baselines and evaluation metrics and compare the results produced by the networks trained using only the original dataset and VAE-enriched dataset with synthetic training data generated by the VAE.

Baselines

We include three frequently used collagen fiber quantification tools, ImageJ ridge detection plugin (Steger, 1998), the CT-FIRE module in CurveAlign (Bredfeldt et al., 2014a; Liu et al., 2020), and the active contour-based toolbox SOAX (Xu et al., 2015, 2014b), in our baselines. These three methods take collagen images and produce coordinate lists that define individual fiber centerlines.

Evaluation metrics

To assess the quality of the fiber extraction results, we use two types of quantitative metrics that serve as similarity measurements between the output of the network/algorithm and the corresponding ground truth annotations made by human experts.

The first metric is an image-level similarity measurement. The 6 collagen fiber properties defined in the method Sec. 3.3, namely fiber circular mean, fiber circular variance, fiber density, fiber waviness, fiber length mean, fiber length variance, are computed from the network/algorithm extracted fiber center-lines and human annotations. The absolute differences of each property are computed for all output-annotation pairs and normalized by dividing the differences using the maximum value of the corresponding property in the annotations. This metric is denoted using the term “normalized error”. A smaller normalized error suggests the result is more similar to the ground truth. We also rank the errors of the methods for the properties (denoted as “rank score”). A method receives 0 points if it produces a property value closest to the ground truth, receives 1 point if it produces the second closest value, and so on. The points are averaged across 6 properties, and a small point (i.e., lower rank score) indicates that the output is closer to the ground truth annotation.

Additionally, we use a soft version of Intersection over Union (soft-IoU) (Máttyus et al., 2017) with a Gaussian filter as a pixel-wise metric. Soft-IoU with a Gaussian filter is less sensitive to pixel-wise shifting than IoU; thus, it is more suitable to quantify the overlap between pairs of skeleton-like structures such as collagen fiber centerlines. Two centerlines that are only one pixel shifted can have a 0 IoU score, but the soft-IoU score remains high. Soft-IoU scores are computed for all output-annotation pairs in the test set and averaged. The centerline extraction network was trained with a batch size 128 for 10k epochs.

Evaluation results

Table 1 summarizes the average normalized error of each property and the average Soft-IoU on the testing set. Fig. 9 shows the violin plots of the values in the 5-95 percentile range for each of the normalized errors and Soft-IoU for testing images. Representative results of input images captured using different objective magnifications (20× and 40×) are shown in Fig. 10. The results show that the centerlines produced by the centerline extraction networks are consistently more similar to the ground truth annotations, with smaller normalized errors and larger soft-IoU to the ground truth. We also noticed that the network is less sensitive to the different lens magnifications utilized (i.e., 20× vs. 40×) while the parameters chosen for ridge detection and SOAX can lead to over/under extraction results for one of the magnifications. The network trained using the VAE-enriched dataset yields better results than the network trained using only the original dataset and has smaller error variances on the test set, indicating better generalization. This result suggests that the VAE is capable of increasing the diversity and variation of the original dataset by generating synthetic training data, and the downstream network trained using the enriched dataset can generalize better on testing samples not seen by the network.

Table 1:

Averaged normalized error of centerline properties, Soft-IoU, and Rank score computed from the centerline extraction results of different methods (includes all data points). For the normalized error (circular mean, circular variance, length mean, and length variance) and rank score, smaller values indicate better results. For Soft-IoU, larger values indicate better results. Network (+VAE) are the results where the network inputs were enriched with VAE generated outputs.

Method	Circular mean↓	Circular variance↓	Length mean↓	Length variance↓	Waviness↓	Fiber density↓	Soft-IoU↑	Rank score↓
Ridge detector	0.0565	0.1424	0.1318	0.1456	0.1772	0.1453	0.6638	2.5922
CT-FIRE	0.0525	0.1201	0.0748	0.0691	0.1897	0.1221	0.6680	2.1184
SOAX	0.0594	0.0969	0.0843	0.1016	0.1706	0.0993	0.6113	2.0129
Network (original)	0.0514	0.0841	0.0623	0.0747	0.2002	0.0908	0.7160	1.8026
Network (+VAE)	0.0524	0.0766	0.0534	0.0672	0.1811	0.0778	0.7207	1.4738

Fig. 9:

The violin plots of the values in the 5-95 percentile range for each of the normalized errors and Soft-IoU, calculated from the outputs of the centerline extraction network (Stage III) and ground truth annotations for individual testing images. The white dots in the plots indicate median values. The 5-95 percentile range is used to exclude outliers and create y-axis ranges that better show the performance differences between the methods.

Fig. 10:

Collagen fiber extraction outputs of different methods. Top row: Outputs for a collagen fiber image captured at 20× magnification. Bottom row: Outputs for a collagen fiber image captured at 40× magnification. Columns from left to right: Input image, manual annotation, outputs of ridge detector (Steger, 1998), outputs of CT-FIRE (Bredfeldt et al., 2014b; Liu et al., 2017, 2020), outputs of the neural network trained using the original dataset, outputs of the neural network trained using VAE-enriched dataset.

5. Discussion and Conclusion

This study demonstrates a workflow for collagen fiber centerline extraction using deep models. To tackle the challenge of creating annotations on a large scale, we propose a synthetic training data generation pipeline powered by VAE. Our proposed VAE framework (DuoVAE) can generate synthetic collagen fiber centerlines that resemble human annotations, and the properties of the generated collagen fiber centerlines can be controlled via an auxiliary branch. Though appropriate regularization coefficients have to be determined experimentally, the implementation of our method is considerably simpler than some recent work for controlled synthetic data generation. By synthesizing realistic collagen fiber images from synthetic centerlines via a cGAN, more image-mask pairs with larger varieties can be generated. The downstream collagen fiber extraction network generalizes better using the enriched dataset for training. Comparing our method to widely used collagen fiber analysis toolboxes shows a clear-cut improvement for collagen fiber centerline extraction in cancer tissue samples. Moreover, evaluations of DuoVAE on two VAE benchmark datasets, dSprites and 3dshapes, show the ability of DuoVAE to disentangle the latent variables for high-quality property-controlled data generation.

We plan to integrate the presented DuoVAE-based workflow into the CurveAlign/CT-FIRE toolbox, which has a user-friendly graphical interface as well as other useful downstream functionalities for collagen fiber analysis in tissue, such as fiber width estimation, local and global fiber density and alignment analysis with respect to tumor boundaries. All of these analyses are benefited from accurate collagen fiber centerline extractions. Future work will explore the impact of additional constraints for representation disentanglement in our DuoVAE architecture, as well as enable the generation of synthetic training data with higher resolution. Furthermore, the current synthetic data generation scheme separates individual training phases for synthetic centerlines and synthetic collagen fiber images. A more ideal solution will be to combine the two phases into a single phase and generate synthetic image-centerline pairs in one training phase that can be optimized end-to-end. This process can possibly benefit from self-supervised pretraining using unannotated collagen images which do not require careful human curation.

• Deep learning-based collagen fiber centerline extraction in cancer tissue

• Synthetic data for training deep neural networks and increasing generalizability

• Variational autoencoder for property-controlled data generation

6. Appendix

6.1. Derivations of ELBO in DuoVAE

6.1.1. VAE and the Evidence Lower Bound (ELBO)

Given a generative model of the form $p_{\theta }(\mathbf{z},\mathbf{x})=p_{\theta }(\mathbf{z})p_{\theta }(\mathbf{x}\mid \mathbf{z})$ where we want to perform posterior inference by computing $p_{\theta }(\mathbf{z}\mid \mathbf{x})=p_{\theta }(\mathbf{z},\mathbf{x})/p_{\theta }(\mathbf{x})$, a VAE approximates the true (unknown) distribution of data $p(\mathbf{x})$ using a parameterized model $p_{\theta }(\mathbf{x})$, which can be achieved by maximizing the ELBO of marginal distributions over some observed data $p_{\theta }(\mathbf{x})=\int p_{\theta }(\mathbf{x},\mathbf{z})d\mathbf{z}=\int p_{\theta }(\mathbf{x}\mid \mathbf{z})p_{\theta }(\mathbf{z})d\mathbf{z}$. The expression for ELBO can be derived from several approaches including via Jensen’s inequality, minimization of KL-divergence between a parametric inference model (i.e., encoder) $q_{\varphi }(\mathbf{z}\mid \mathbf{x})$ and the intractable posterior $p_{\theta }(\mathbf{z}\mid \mathbf{x})$, or by decomposing the log-likelihood $\log p_{\theta }(\mathbf{x})$ as

$$\begin{gathered} \text{log}{p}_{\theta }(x)={𝔼}_{q_{\varphi }(z\mid x)}[\text{log}{p}_{\theta }(x)] \\ ={𝔼}_{q_{\varphi }(z\mid x)}\left[\text{log}\left(\frac{p_{\theta }(x,z)}{p_{\theta }(z\mid x)}\frac{q_{\varphi }(z\mid x)}{q_{\varphi }(z\mid x)}\right)\right] \\ ={𝔼}_{q_{\varphi }(z\mid x)}\left[\text{log}\frac{q_{\varphi }(z\mid x)}{p_{\theta }(z\mid x)}\right]+{𝔼}_{q_{\varphi }(z\mid x)}\left[\text{log}\frac{p_{\theta }(x\mid z)p_{\theta }(z)}{q_{\varphi }(z\mid x)}\right] \\ ={𝔼}_{q_{\varphi }(z\mid x)}\left[\text{log}\frac{q_{\varphi }(z\mid x)}{p_{\theta }(z\mid x)}\right] \\ +{𝔼}_{q_{\varphi }(z\mid x)}[\text{log}{p}_{\theta }(x\mid z)]-{𝔼}_{q_{\varphi }(z\mid x)}\left[\text{log}\frac{q_{\varphi }(z\mid x)}{p_{\theta }(z)}\right] \\ =D_{KL}\left(q_{\varphi }(z\mid x)\parallel p_{\theta }(z\mid x)\right) \\ +\underset{=ℒ(x)(\text{ELBO})}{\underbrace{{𝔼}_{q_{\varphi }(z\mid x)}[\text{log}{p}_{\theta }(x\mid z)]-D_{KL}\left(q_{\varphi }(z\mid x)\parallel p_{\theta }(z)\right)}}\ge ℒ(x). \end{gathered}$$ (8)

Due to the non-negativity of the KL-divergence $D_{KL}\left(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\parallel p_{\theta }(\mathbf{z}\mid \mathbf{x})\right)\ge 0$, estimating $p_{\theta }(\mathbf{x})$ can be viewed as maximizing the ELBO $ℒ(\mathbf{x})$ w.r.t. the variational parameters $\theta$ and $\varphi$, which simultaneously minimizes $D_{KL}\left(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\parallel p_{\theta }(\mathbf{z}\mid \mathbf{x})\right)$ and yields the approximate posterior $q_{\varphi }(\mathbf{z}\mid \mathbf{x})$.

6.1.2. ELBO of the main VAE

Given the generative process of the main VAE $p_{\theta }(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{w})=p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{w})p_{\theta }(\mathbf{z})p_{\theta }(\mathbf{w}\mid \mathbf{y})p_{\theta }(\mathbf{y})$ and the factorized approximate posterior $q_{\varphi }(\mathbf{z},\mathbf{w}\mid \mathbf{x})=q_{\varphi }(\mathbf{z}\mid \mathbf{x})q_{\varphi }(\mathbf{w}\mid \mathbf{x})$, the ELBO of the log-likelihood $\log p_{\theta }(\mathbf{x},\mathbf{y})$ is expressed as

$$\begin{gathered} \text{log}{p}_{\theta }(x,y)\ge {𝔼}_{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}\left[\text{log}\frac{p_{\theta }(x,y,z,w)}{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}\right] \\ ={𝔼}_{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}\left[\text{log}\frac{p_{\theta }(x\mid z,w)p_{\theta }(z)p_{\theta }(w\mid y)p_{\theta }(y)}{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}\right] \\ ={𝔼}_{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}[\text{log}{p}_{\theta }(x\mid z,w)] \\ -{𝔼}_{q_{\varphi }(w\mid x)q_{\varphi }(z\mid x)}\left[\text{log}{q}_{\varphi }(z\mid x)-\text{log}{p}_{\theta }(z)\right] \\ -{𝔼}_{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}\left[\text{log}{q}_{\varphi }(w\mid x)-\text{log}{p}_{\theta }(w\mid y)\right]+\text{log}{p}_{\theta }(y) \\ \propto {𝔼}_{q_{\varphi }(z\mid x)q_{\varphi }(w\mid x)}[\text{log}{p}_{\theta }(x\mid z,w)]-{\beta }_z{D}_{KL}\left(q_{\varphi }(z\mid x)\parallel p_{\theta }(z)\right) \\ -D_{KL}\left(q_{\varphi }(w\mid x)\parallel p_{\theta }(w\mid y)\right)+\text{log}{p}_{\theta }(y)=:{\overline{ℒ}}_1(x,y), \end{gathered}$$ (9)

where we introduced the regularization coefficient ${\beta }_z$ similar to $\beta$-VAE (Higgins et al., 2017) to control the capacity of the latent information in $\mathbf{z}$.

6.1.3. ELBO of the auxiliary VAE

Given the observed property values $\mathbf{y}$, the objective of the auxiliary VAE is to maximize the log-likelihood $p_{\theta }(\mathbf{y})$, which, in turn, simultaneously approximates the intractable posterior $p_{\theta }(\mathbf{w}\mid \mathbf{y})$ in Eq. 9 with an approximate posterior $q_{\delta }(\mathbf{w}\mid \mathbf{y})$ (i.e., encoder). This can be achieved by maximizing the ELBO ${ℒ}_2(\mathbf{y})$ which can be derived using the same approach in Eq. 8, written as

$$\begin{gathered} \text{log}{p}_{\theta }(y)={𝔼}_{q_{\delta }(w\mid y)}[\text{log}{p}_{\theta }(y)] \\ ={𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}\left(\frac{p_{\theta }(y,w)}{p_{\theta }(w\mid y)}\frac{q_{\delta }(w\mid y)}{q_{\delta }(w\mid y)}\right)\right] \\ ={𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}\frac{q_{\delta }(w\mid y)}{p_{\theta }(w\mid y)}\right]+{𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}\frac{p_{\gamma }(y\mid w)p_{\theta }(w)}{q_{\delta }(w\mid y)}\right] \\ ={𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}\frac{q_{\delta }(w\mid y)}{p_{\theta }(w\mid y)}\right] \\ +{𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}{p}_{\gamma }(y\mid w)\right]-{𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}\frac{q_{\delta }(w\mid y)}{p_{\theta }(w)}\right] \\ \propto D_{KL}(q_{\delta }(w\mid y)\parallel p_{\theta }(w\mid y)) \\ +\underset{={ℒ}_2(y)(\text{ELBO})}{\underbrace{{𝔼}_{q_{\delta }(w\mid y)}\left[\text{log}{p}_{\gamma }(y\mid w)\right]-{\beta }_w{D}_{KL}(q_{\delta }(w\mid y)\parallel p_{\theta }(w))}}\ge {ℒ}_2(y), \end{gathered}$$ (10)

where we introduced a regularization coefficient ${\beta }_w$ similar to $\beta$-VAE (Higgins et al., 2017).

6.2. Additional derivations

Our generative process is defined as $p(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{w})=p(\mathbf{x},\mathbf{y}\mid \mathbf{z},\mathbf{w})p(\mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})p(\mathbf{w}\mid \mathbf{y})p(\mathbf{y})p(\mathbf{z})$ which also implies $p(\mathbf{x},\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})p(\mathbf{y}\mid \mathbf{w})$. In this section, we justify $p(\mathbf{x},\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})p(\mathbf{y}\mid \mathbf{w})$ by first showing (1) $p(\mathbf{x},\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})p(\mathbf{y}\mid \mathbf{z},\mathbf{w})$, and then (2) $p(\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{y}\mid \mathbf{w})$ given the following two assumptions:

1. information on $\mathbf{y}$ is encoded to $\mathbf{w}$; and, $\mathbf{x}$ and $\mathbf{y}$ are conditionally independent given $\mathbf{w}$ (i.e., $\mathbf{x}\perp \mathbf{y}\mid \mathbf{w}$),

2. $\mathbf{z}$ is independent of $\mathbf{y}$ and $\mathbf{w}$ (i.e., $\mathbf{z}\perp \mathbf{y}$ and $\mathbf{z}\perp \mathbf{w}$), which gives $\mathbf{y}\perp \mathbf{z}\mid \mathbf{w}$ (Derivation in Sec. 6.2.2).

6.2.1. Derivation of $p(\mathbf{x},\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})p(\mathbf{y}\mid \mathbf{z},\mathbf{w})$

We show that if $\mathbf{z}\perp \mathbf{y},\mathbf{z}\perp \mathbf{w}$, and $\mathbf{x}\perp \mathbf{y}\mid \mathbf{w}$, then $\mathbf{x}\perp \mathbf{y}\mid (\mathbf{z},\mathbf{w})$, which gives us $p(\mathbf{x},\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})p(\mathbf{y}\mid \mathbf{z},\mathbf{w})$. First, according to Bayes’ theorem,

$$p(x,y\mid z,w)=\frac{p(x,y,z,w)}{p(z,w)}\frac{p(x,z,w)}{p(x,z,w)}=p(y\mid x,z,w)p(x\mid z,w),$$ (11)

and

$$p(x,y\mid z,w)=\frac{p(x,y,z,w)}{p(z,w)}\frac{p(y,z,w)}{p(y,z,w)}=p(x\mid y,z,w)p(y\mid z,w).$$ (12)

Table 2:

DuoVAE architecture for evaluations on dSprites and 3dshapes datasets (Sec. 6.3).

EncoderX	DecoderX	EncoderY{1,2,3}	DecoderY{1,2,3}
Conv2d(c, 32, 4, 2)-BN-ReLU	Linear(z-dim + w-dim, 128)-ReLU	Linear(1, 8)-ReLU	Linear(1, 8)-ReLU
Conv2d(32, 32, 4, 2)-BN-ReLU	Linear(128, 128)-ReLU	Liner(8, 2)	Linear(8, 1)-Sigmoid
Conv2d(32, 32, 4, 2)-BN-ReLU	Linear(128, 2048)-ReLU
Linear(2048, 128)-ReLU	ConvTranspose2d(32, 32, 4, 2)-BN-ReLU
Linear(128, 128)-ReLU	ConvTranspose2d(32, 32, 4, 2)-BN-ReLU
Linear(128, 2 × (z-dim + w-dim))	ConvTranspose2d(32, 32, 4, 2)-BN-ReLU
	Conv2d(32, c, 4, 2)

Table 3:

DuoVAE hyperparameters for evaluations on dSprites and 3dshapes datasets (Sec. 6.3).

Learning rate	Batch size	dimension of z (z-dim)	dimension of w (w-dim)	${\beta }_z$	${\beta }_w$	$\alpha$
0.0005	4096	4	3	3.4	3.4	1

Fig. 11:

Representative latent variable traversing by PCVAE on the supervised latent variables $\mathbf{w}$ (top 3 rows) and the rest of the latent variables $\mathbf{z}$ (bottom 4 rows).

Since $\mathbf{y}\perp \mathbf{x}\mid \mathbf{w}(\iff \mathbf{x}\perp \mathbf{y}\mid \mathbf{w})$, as well as $\mathbf{z}\perp \mathbf{y}$ and $\mathbf{z}\perp \mathbf{w}$ (i.e., $\mathbf{y}\perp \mathbf{z}\mid \mathbf{w}$ with derivation in Sec. 6.2.2), we have $p(\mathbf{y}\mid \mathbf{x},\mathbf{z},\mathbf{w})=p(\mathbf{y}\mid \mathbf{w})$ and $p(\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{y}\mid \mathbf{w})$ based on conditional independence in probability theory. Substituting them to Eq. 11 and 12 gives $p(\mathbf{x}\mid \mathbf{y},\mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})$, i.e., $\mathbf{x}\perp \mathbf{y}\mid (\mathbf{z},\mathbf{w})$. Therefore, since $\mathbf{x}\perp \mathbf{y}\mid (\mathbf{z},\mathbf{w})$, we can have $p(\mathbf{x},\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{x}\mid \mathbf{z},\mathbf{w})p(\mathbf{y}\mid \mathbf{z},\mathbf{w})$ (∵ conditional independence).

6.2.2. Derivation of $p(\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{y}\mid \mathbf{w})$

We show that if $\mathbf{z}\perp \mathbf{y}$ and $\mathbf{z}\perp \mathbf{w}$, then $\mathbf{y}\perp \mathbf{z}\mid \mathbf{w}$, which gives us $p(\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{y}\mid \mathbf{w})$. According to Baye’s theorem,

$$p(y,z\mid w)=\frac{p(y,z,w)}{p(w)}\frac{p(y,w)}{p(y,w)}=p(z\mid y,w)p(y\mid w),\text{and}$$ (13)

$$p(y,z\mid w)=\frac{p(y,z,w)}{p(w)}\frac{p(z,w)}{p(z,w)}=p(y\mid z,w)p(z\mid w).$$ (14)

Since $\mathbf{z}\perp \mathbf{y}$ and $\mathbf{z}\perp \mathbf{w}$, we have $p(\mathbf{z}\mid \mathbf{y},\mathbf{w})=p(\mathbf{z})$ and $p(\mathbf{z}\mid \mathbf{w})=p(\mathbf{z})$. Substituting them to Eq. 13 and 14 gives $p(\mathbf{y}\mid \mathbf{z},\mathbf{w})=p(\mathbf{y}\mid \mathbf{w})$, i.e., $\mathbf{y}\perp \mathbf{z}\mid \mathbf{w}$.

6.3. Architecture and hyperparameters for dSprites and 3dshapes datasets

The neural network architecture (Table 2) for DuoVAE consists of EncoderX to model $q_{\varphi }(\mathbf{z},\mathbf{w}\mid \mathbf{x})$, DecoderX to model $p_{\varphi }(\mathbf{x}\mid \mathbf{z},\mathbf{w})$, EncoderY₁, EncoderY₂, and EncoderY₃ to model $q_{\delta }\left(w_1\mid y_1\right),q_{\delta }\left(w_3\mid y_3\right)$, and $q_{\delta }\left(w_3\mid y_3\right)$, respectively, and similarly DecoderY₁, DecoderY₂, and DecoderY₃ to model $p_{\gamma }\left(y_1\mid w_1\right),p_{\gamma }\left(y_2\mid w_2\right)$, and $p_{\gamma }\left(y_3\mid w_3\right)$, respectively. Each set of layers of the network is expressed as Conv*(input channel, output channel, kernel size, stride) for convolutional layer, Linear(input channel, output channel) for fully-connected layer, and BN and ReLU denote batch normalization across mini-batches and rectified linear unit function activation layer, respectively. We use the same architecture and hyperparameters for both the training on dSprites and 3dshapes datasets, respectively. The number of input/output channels $c$ is 1 for dSprites (grayscale) and 3 for 3dshapes (RGB). The hyperparameters (Table 3) include ${\beta }_z,{\beta }_w$, and $\alpha$ as introduced in Sec. 3.1.

6.4. Additional Evaluation and Comparison for property-controlled data generation

6.4.1. dSprites dataset

Representative visualization of the latent variables traversing by PCVAE can be found in Fig. 11, in addition to the DuoVAE’s in Fig. 8. The same traverse setting as DuoVAE is used. The first 3 rows correspond to object size, object horizontal location, and object vertical location. Additional rotation effects are observed when the horizontal and vertical locations are changed. Whereas in 8, the results of DuoVAE show more clean control on the object locations.

Fig. 12:

Representative latent variable traversing by (a) DuoVAE and (b) PCVAE. For (a) DuoVAE, each of the supervised latent variables $\mathbf{w}$ (top 3 rows) captures the information of (scale, wall color, floor color), respectively, whereas the rest of the latent variables $\mathbf{z}$ (bottom 4 rows) appear to have captured the rest of the information (e.g., rotation, shape, and object color) in an entangled way.

Fig. 13:

The heatmaps of the normalized mutual information between each of the properties (scale, wall color, floor color) and latent variables $\left(w_1,w_2,w_3\right)$ for 3dshapes dataset. DuoVAE attains higher mutual information scores compared to PCVAE, which suggests that the performance of DuoVAE is comparable to the other recent models, though having relatively simpler architecture and implementation.

6.4.2. 3dshapes dataset

Similar to the quantitative evaluation results on the dSprites dataset in Fig. 7–8, we provide the heatmaps of normalized mutual information (Fig. 13) and the visualizations of latent variable traversing (Fig. 12) on the 3dshapes dataset for DuoVAE and PCVAE. The first 3 rows correspond to the size of the cube, the color of the wall, and the color of the floor. It can be seen that in PCVAE, the color of the cube is affected when the floor color changes, whereas in DuoVAE the color of the cube stays almost unchanged.

6.4.3. Ablation study

Hyperparameter tuning.

We investigate the impacts of the weight coefficients ${\beta }_z$, and ${\beta }_w$ in the loss functions of DuoVAE (Eq. 2–3) on the performance of property-controlled data generation. We trained DuoVAE on 31k images of dSprites dataset randomly generated from uniformly sampled four independent latent properties: scale, x-position, y-position, and orientation. We supervised three of the properties $\left\{y_1,y_2,y_3\right\}$ for scale, x-position, y-position, respectively, and left orientation unsupervised. We experimented with ${\beta }_w=\{\mathrm{0.1,1},\mathrm{10,1000}\}$ and ${\beta }_z=\{\mathrm{0.1,1},10\}$, conducting 12 independent trainings. The architecture used throughout the ablation study is provided in Table 4.

Table 4:

DuoVAE architecture for the ablation study on a subset of dSprites dataset.

EncoderX	DecoderX	EncoderY{1,2,3}	DecoderY{1,2,3}
Conv2d(1, 32, 4, 2)-BN-ReLU	Linear(6, 32)-ReLU	Linear(1, 8)-ReLU	Linear(1, 8)-ReLU
Conv2d(32, 32, 4, 2)-BN-ReLU	Linear(32, 32)-ReLU	Liner(8, 2)	Linear(8, 1)-Sigmoid
Conv2d(32, 32, 4, 2)-BN-ReLU	Linear(32, 2048)-ReLU
Linear(2048, 32)-ReLU	ConvTranspose2d(32, 32, 4, 2)-BN-ReLU
Linear(32, 32)-ReLU	ConvTranspose2d(32, 32, 4, 2)-BN-ReLU
Linear(32, 12)	ConvTranspose2d(32, 32, 4, 2)-BN-ReLU
	Conv2d(32, 1, 4, 2)

Fig. 14:

Data generation while traversing three properties {y₁, y₂, y₃} = {scale, x-position, y-position}, one at a time for each of the 12 parameters sets $\left({\beta }_w,{\beta }_z\right)$.

In the generation process, latent variables $\mathbf{z},\mathbf{w}\in {ℝ}^3$ are encoded using the encoder of the main VAE given a test image $\mathbf{x}$; then, each of the ground truth properties $\mathbf{y}\in {ℝ}^3$ is traversed one at a time from its minimum to maximum values while keeping $\mathbf{z}$ constant. For each generation, the encoder of the auxiliary VAE encodes a new set of latent variables ${\mathbf{w}}^{\prime }\in {ℝ}^3$ from different values of $\mathbf{y}$. Then, the decoder of the main VAE generates a new image from $\left(\mathbf{z},{\mathbf{w}}^{\prime }\right)$. All models were trained for 4k epochs with a learning rate of 0.001 and batch size of 1024. Moreover, $\alpha =5$ (Eq. 4) was used for all models.

The result is shown in Fig. 14. Each of the 12 sub-panels consists a set of $\left\{y_1,y_2,y_3\right\}$ traverses corresponding to a combination of $\left({\beta }_w,{\beta }_z\right)$ whose values are provided in the $y$ and $x$ axis label, respectively. Hypothetically, the three properties scale, x-position, y-position can be controlled via the supervised variables $\left\{y_1,y_2,y_3\right\}$, respectively, while the latent variables $\mathbf{z}$ capture all the other information (e.g., orientation).

For $\beta$-VAE (Higgins et al., 2017), the hyperparameter $\beta$ balances latent channel capacity and reconstruction fidelity. Similarly for DuoVAE, ${\beta }_w$ and ${\beta }_z$ create a trade-off between the data generation controllability and fidelity. For larger values of ${\beta }_w$, the data controllability is presumably decreased as the result of a decreased latent channel capacity of $\mathbf{w}$; this is shown in the sub-panels in the columns of ${\beta }_w=10$ and ${\beta }_w=100$ where traversing of $y_2$ and $y_3$ did not effectively change the $x$ and $y$ positions, respectively. Moreover, data fidelity is decreased for larger values of ${\beta }_z$ by which the generated images in the row of ${\beta }_z=10$ appear more blurry than the rows of ${\beta }_z=1$ and ${\beta }_z=0.1$. Note that both the data generation controllability and fidelity deteriorate for large values of ${\beta }_w$ and ${\beta }_z$ as shown in the sub-panel at $\left({\beta }_w,{\beta }_z\right)=(\mathrm{100,10})$. The optimal values of ${\beta }_w$ and ${\beta }_z$ were both around or slightly larger than 1, which achieved both the data generation controllability and fidelity as shown in the sub-panel at $\left({\beta }_w,{\beta }_z\right)=(\mathrm{1,1})$.

Data augmentation ratio.

We present the impacts of different ratios of synthetic training on the final fiber centerline extraction performance. The same augmentation procedure in Sec. 4.1 was used for synthesizing the training data and generating 1,188, 4,752, and 10,692 synthetic crops, corresponding to augmented-original data ratios of 2×, 5×, and 10×, respectively, combined with the original 1,188 annotated based crops. The centerline extraction network was trained using each of the original and augmented datasets separately and evaluated on the 199 test crops described in Sec. 4.3.

The evaluation result is shown in Fig. 15, where the dots indicate the means of values in the 5-95 percentile range, and the downward/upward arrows indicate whether the smaller/larger metric is better for each property. In general, the centerline extraction results are improved when more synthetic training data is included. However, each training epoch will take a longer time to finish due to the increased amount of training data. Among the properties of interest, circular variance, density, waviness, fiber length variance, and soft-IoU benefit more from the augmented training data. This is consistent with the observation mentioned earlier since these properties have relatively large error scales. Considering a trade-off between training time and performance, an augmentation ratio of 5× was used in the comparisons to the baselines.

Fig. 15:

The evaluation result of the collagen fiber centerline extraction network (Stage III) trained on the original 1,188 annotated base crops combined with 2×, 5×, and 10× augmented synthetic training crops, respectively. The dots indicate the means of values in the 5-95 percentile range, and the downward/upward arrows indicate whether the smaller/larger metric is better for each property. All models were trained with batch size 64 for 20k epoch