Artificial Second Harmonic Generation Image Feature Tuning with Generative Models and SeFa Semantic Analysis

Abstract

Collagen fibers within the extracellular matrix undergo morphological changes in the presence of many diseases including cancers and connective tissue disorders. Second Harmonic Generation (SHG) microscopy is ideal for visualizing this morphology by its intrinsic contrast and 3D imaging capabilities of fibrillar collagen. To build upon the success of deep learning-based methods analyzing microscopy datasets, in this study, we adapted the generative adversarial network model StyleGAN specifically for use with SHG data. Here we trained our model StyleGAN2-ADA-SHG on a dataset of 1,319 SHG images to extract further information content. When comparing evaluation metrics between the baseline dataset and the generator, we found that the model is well suited to generating novel images that appear perceptually similar to those from our small set of training data. Using Semantic Factorization (SeFa) on the generated images to examine the 10 highest scoring eigenvectors, we found two semantics with strong correlation to fiber morphological features that were quantified via analysis using CT-Fire, CurveAlign, and other fiber-based metrics. We integrated analysis tools into the SeFa GUI to support an extensible framework for future analyses and image editing. Additionally, we tested two image projection methods for editing SHG images in the GAN’s latent space using the identified semantics. The encoder-based approach significantly outperformed the Learned Perceptual Image Patch Similarity (LPIPS) method. With further refinement, these tools can be used to create SHG images of collagen fibers with variable morphology based on disease types for additional analysis purposes.

1. Introduction

Fibrillar collagen serves a key role in structural organization, especially within the extracellular matrix (ECM). The mechanical and structural properties of collagen architecture are altered in many diseases including cancers, fibroses, and connective tissue disorders (1–3). Due to its sensitivity/specificity, Second Harmonic Generation (SHG) microscopy has emerged as a powerful tool to visualize collagen morphology in a wide range of these and other diseased states. Analysis of this morphology has the potential to improve understanding of disease advancement and may, in future studies, contribute to earlier detection strategies (2).

We have investigated the application of machine learning techniques to classify alterations from normal collagen structure in several diseased states. We previously used “textons” (or texture features) to classify a series of tissues related to ovarian cancer. Texture features were identified using image responses to sets of filter banks, and a representative model was generated for specific tissue types based on the distribution of the respective features in each class (4). Ultimately, this method was able to achieve a classification accuracy >90% using one versus the rest classification. We also used principal component analysis of 2D wavelet transforms of idiopathic pulmonary fibrosis and classified normal versus fibrotic disease with >95% accuracy. Unfortunately, while these classification methods resulted in high accuracy, they cannot determine the most important and differentiating aspects of each class of tissue. Our two overarching goals here are to: (i) Learn the most characteristic features of a disease class, and (ii) then use SHG images as blueprints for fabrication of tissue engineered scaffolds to study the cell biology of disease progression (5). To model and study such behavior, it is essential to go beyond classification – we seek to develop a high throughput pipeline to optimize biomimetic model design, fabrication, and testing. To this end, we describe how generative models trained specifically on SHG data can help in exploring and editing the morphological features of synthetically generated images of collagen fibers.

In recent years, GANs have been used in an increasingly broad range of applications, including specific use cases in microscopy. However, training GANs with small sample sizes remains challenging. Even in settings where sufficient data in terms of sample sizes is available, it is not always usable, where for example, in some modalities, resolution can be insufficient, or images can be noisy. But GANs, once successfully trained, can be used to enhance images (e.g. sharpness, noise, and clarity). By training GANs on labeled data, they can be used to automatically segment datasets by identifying characteristic features in diseased tissues, including tumors (6, 7). In addition to these tasks, microscopy modalities can use GANs for virtual staining, generating realistic H&E, Masson’s Trichrome, and Jones’ images from unlabeled data. Similarly, digital fluorescence labeling can be performed on brightfield or phase contrast images (8, 9). These and other implementations have achieved strong results using models based on StyleGAN, CycleGAN, SPADE GAN, and HingeGAN (10) architectures. Typically, these generators are evaluated based on the Fréchet inception distance (FID), which measures the feature vector distance between synthetic and real images. Style-based GANs have achieved particularly good FIDs, indicating successful generation of quality image data reflective of that available in microscopy (11). Importantly, Style-GANs have been used to extract image features and assess their editability, where latent feature analysis has been applied to human faces where the layers correspond to examples such as head shape, hair style, and color scheme (12). To our knowledge, latent space feature analysis and editing within the generator for microscopy datasets has remained somewhat limited so far, where this represents a new opportunity for analysis.

Our model is based on StyleGAN2-ADA architecture, where this was chosen in large part due to its broad use in the computer vision community and empirical performance relative to other backbones in our initial testing. The latent space is also accessible, which makes it well suited for our use case. Briefly, StyleGAN allows the generator network to alter specific features of the image through the use of an intermediate latent space compared to the more commonly used Gaussian space. The latent space (for deep generative models, but also for deep models in general) is typically high-dimensional. It allows us to interpret representations or embeddings of the data (e.g., images) learned by the model (13) and uses so-called style blocks empirically. By encouraging individual style blocks to behave “independently”, StyleGAN can achieve a more linear and disentangled latent space (14). Also, layers in the model network often correspond to different image styles, which consequently allows easier image editability (15). When StyleGAN is trained on SHG images, we expect the styles to correspond to different features of collagen fibers such as density, length, width, curvature, and alignment (16). StyleGAN2-ADA, is specifically well-suited to microscopy images because it uses adaptive discriminator augmentation to train with limited data, dynamically adjusting input into the generator based on the degree of overfitting, greatly improving training convergence for limited datasets such as those of human and animal tissues (17).

By adapting StyleGAN for SHG images, (we dub StyleGAN2-ADA-SHG) and training on a database of SHG images of both normal and diseased tissue, we generated latent codes that model features for collagen networks. Analyzing the generator’s affine layers using an eigenvalue decomposition with SeFa (18), we found interpretable editing directions. Real images can then be encoded and edited along latent directions that correspond to both known and unknown features that reflect disease states. These editable latent directions are often referred to as semantics. As a proof-of-concept, this approach establishes a pipeline that could in future work create more accurate biomimetic models and generate models for disease progression. It could also alleviate unique difficulties faced in small microscopy-based training sets (19). With further development and sufficient data, we could also include disease-stratified training and validation. This would better identify and assess structural changes between normal tissues and across stages of cancers (e.g. ovarian), where this could contribute to a better understanding of tumor progression. Here, we use external analysis tools for collagen fiber features to quantify the behavior of the GAN-based learned semantics.

2. Methods

2.1. Sample Library

SHG images (curated from both published and unpublished work) were previously collected from several collagen-rich tissues using our established imaging protocols (20). All images were acquired at an excitation wavelength of 890 nm with a field of view of 180 × 180 μm; laser power and additional imaging parameters varied by tissue type and are reported in Supplementary Table S.1. The open-source ImageJ platform for image analysis (FIJI) was used to process all images. A custom macro was built to transform raw tiff images to 8-bit greyscale 512×512 PNG images. Images were manually curated for inclusion based on resolution, fiber clarity (i.e., differentiated fiber structure rather than diffuse SHG signal), and absence of confounding signals such as muscle tissue, tendon, or hair. The final dataset comprises 1,319 images.

A set of images from an older version of the microscope software had artifacts on the brightest fibers, affecting only the ovarian cancer (C-series) subset (254 images, 19.3% of the dataset). Custom Matlab2023a code was used to correct these artifacts using a nearest-neighbor pixel by pixel inpainting approach: artifact pixels were identified by thresholding dark pixels within bright fiber regions, and masked areas were restored using local texture from surrounding valid pixels (Supplementary Figure S.1). To validate that these artifacts did not propagate to the generator, 100 randomly generated images were manually inspected. Although the original artifact was not observed, two of the generated images showed a distinct smearing artifact (Supplementary Figure S.2).

2.2. Generative Adversarial Network

The StyleGAN2 algorithm is described in detail in Karras et al (15). Briefly, a GAN consists of a generator and discriminator network that work in conjunction to create a distribution of synthetic images that will ideally be statistically indistinguishable from the distribution of the real training set. It is the discriminator’s role to check whether the two distributions (synthetic versus real) are indistinguishable. In StyleGAN, the discriminator is a progressively grown network (21). The generator is broken into two parts: the mapping network f, which inputs a latent vector z ε Ƶ onto the auxiliary latent space W, and the synthesis network g, which maps vector w ε W to the output image. The distribution of the latent codes in W is learned by mapping network f, which allows for a more disentangled latent space than in Ƶ space. The synthesis network consists of a series of style blocks that each consist of modulation, demodulation (normalization), and convolution. Biases and noise broadcast operation B are added outside the active area of the style so that they affect only normalized data (Figure 1).

Figure 1.

The StyleGAN2 architecture consists of the latent code z from the input space Ƶ which is mapped to a network (f) of fully connected (FC) layers. It is then mapped to an intermediate latent space *W*. In the generator network, “A” is a learned affine transformation, and “B” is a learned per-channel scaling factor to add noise. The style weights are within each w, the bias factors are b, and the constant input is c. Each style block consists of modulation, demodulation (normalization), and convolution.

Training in StyleGAN2 is further improved with adaptive discriminator augmentation (ADA). ADA was developed for small training sets that are augmented by performing image alterations to increase training data in order to improve results. This approach dynamically tunes image augmentation with two overfitting heuristics. Changing the probability of applying an image filter (e.g. rotation, distortion, cutouts, etc.) based on estimating what portion of the training set will get positive discriminator outputs significantly reduces the chance of overfitting in the generator model (17).

Our modified model, StyleGAN2-ADA-SHG, was implemented in Pytorch with several alterations from the original code. For example, all inputs, layers, and outputs were changed to greyscale in order to accept SHG images. This changes the input and output size to 512 × 512 × 1, which has the downstream effect of reducing the network dimensionality. Training was conducted using 1319 SHG images, an image batch size of 16, a learning rate of 0.002, and the Adam optimizer. All machine learning training was performed on a GeForce RTX 3080 GPU with 10GB of vRAM. Training time was approximately 48 hrs, including resumed checkpoint runs, and convergence was assessed by monitoring the FID-50k curve for plateau behavior (Figure 5).

Figure 5.

Training curve for Fréchet Inception Distance 50K as scored by StyleGAN2-ADA-SHG’s internal metrics.

2.3. Latent Space Editing with SeFa

We combined a customized version of semantic factorization (SeFa) with the StyleGAN2-ADA model along with additional GUI tools for analytics and visualization. To identify interpretable directions in the latent space of a GAN model, SeFa computes the eigenvectors of the affine transformation A within each style modulation that maps latent codes to modulation parameters. Finding the directions that cause large variation after the projection of A, yields information regarding image variation within the GAN model. The eigenvectors therein are then referred to as semantic directions that traverse the latent space of the model (18). This applies to StyleGAN in the transformation from latent code to style code and can interpret all or any subset of layers. The layer subsets follow the standard architectural divisions defined by Karras et al. (15) for style blocks in StyleGAN: early levels (layers 0–2) correspond to coarse spatial resolution and control high-level features, middle layers (3–4) correspond to smaller-scale features, and end layers (5–7) correspond to higher resolution control such as pixel texture and microstructures. These subsets are predefined in the GUI and were not selected post-hoc.

The factorization yielded a set of ranked eigen-directions, each corresponding to a potential semantic variation. For subsequent analyses, we selected the top 10 components by eigenvalue magnitude.

2.4. CT-FIRE and Image Analysis

To facilitate quantitative analysis of results from the model described above, we needed to establish a reference for comparison. We used the curvelet transform-fiber extraction (CT-FIRE) analysis (22) to determine the length, width, and straightness of the collagen fibers in the generated SHG images. This analysis first uses the fiber extraction (FIRE) algorithm (23) to denoise images, enhance fiber edge features, and extract fibers, followed by the curvelet transform (CT) to determine fiber metrics. FIJI was also utilized in the two-dimension fast Fourier transform (2D-FFT) analysis, where two custom macros utilized the Radial Profile Extended (Philippe Carl, version 2017/04/18) and the Oval Profile Plot (Bill O’Connell, version 2012/03/01) plugins to model collagen fiber alignment (16).

2.5. Image Projection Strategies

Two strategies were implemented for projecting real and GAN images into the latent space of the generator: learned perceptual image patch similarity (LPIPS-based optimization) and a custom-trained encoder. For the optimization-based method, we implemented a projector that minimizes a combination of pixel-wise mean squared error (MSE) LPIPS loss between the input image and a generated image that is synthesized from the latent code w (24). The LPIPS loss was computed using a neural network, VGG, backbone (25). An initial latent vector was selected from 10 random z samples by choosing the one whose mapped w yielded the lowest MSE. The optimization ran for 1000 steps with a learning rate of 0.01. A small amount of noise regularization was applied to suppress artifacts (see projector.py, available on GitHub for full details).

The encoder was adapted from the architecture used in a Stylegan2 pytorch encoder and modified for single-channel input and output (Figure 2) (26). To simplify version integration, optimization was performed with adaptive moment estimation (Adam) and MSE loss (27). Training was performed using only images synthesized from the generator. The encoder was trained with GAN images from the model for 300 epochs with a batch size of 4 and a learning rate of 5e-5.

Figure 2.

Encoder architecture for Style GAN-ADA-SHG derived from the work in (26). The encoder consists of an initial convolution layer with a fused LeakyReLU activation. Stacks of ResBlock are added based on the dimensionality of the images. In this case, there are 7 total blocks consisting of an initial convolution without downsampling, then one with downsampling, where both use LeakyReLU activations. The final output is fed into an EqualConv2d layer that flattens the projection from spatial to latent space. The output is then reshaped to transfer latent codes to the GAN. Architecture was generated with code available on GitHub: https://github.com/HarisIqbal88/PlotNeuralNet.

2.6. Interface Integration and Traversal Evaluation

To inspect the effect of latent semantics on the generated samples, we integrated the encoder and projector, adding fine control to SeFa latent directions into a Streamlit GUI. Users can select among the top 10 directions, adjust weights, and immediately visualize the resulting image modifications. As an initial screening step, a grid was generated with 10 samples per semantic at weights of −3 and +3. This small sample size was designed to detect only large effects (Cohen’s d > 1.0) to identify which semantics warranted further characterization; it was not intended as a definitive statistical test. We note that with 10 semantics tested, the initial p-value threshold of 0.05 does not survive Bonferroni correction (α = 0.005). All subsequent analyses used 50 randomly generated images traversed across 21 weight levels (1,050 observations per semantic), providing robust statistical power for confirmation.

An additional tool was implemented to create customizable traversals of selected semantics. For this analysis, traversals were generated from weights −10.0 to +10.0 along a selected latent direction in increments of 0.1. Images were synthesized using latent vectors offset from the original by scalar multiples of the direction vector. These traversals were used to qualitatively evaluate the controllability and entanglement of semantic edits. Code for the interface is available at: https://github.com/Campagnola-BME-Lab/sefa_collagen

Further analysis on selected semantics used 50 random generated images traversed in increments of one. These samples were then analyzed with CT-Fire to measure all relevant features. To assess the correlation between semantic weight and fiber features, while controlling for other feature variations, partial distance correlation (PDC) was implemented. This approach allows for conditional dependence which effectively captures the relationship between two variables after removing other effects (14). The PDC implementation was adapted from the code available at: github.com/zhenxingjian/Partial_Distance_Correlation

2.7. Network Evaluation Metrics

StyleGAN2-ADA-SHG and the encoder were both evaluated via FID and Kernel Inception Distance (KID). The FID score compares the mean and covariance of real and generated image features by assuming their distribution to be a multivariate Gaussian. A lower score indicates generated images are more similar to the real ones in their distribution (28). KID uses maximum mean discrepancy along with a polynomial kernel to compare real and generated images. As the score approaches zero the generated images are expected to match the real images. Both FID and KID use InceptionV3 as their feature extractor. This network is trained on ImageNet, a dataset of 1.2 million RGB images with labels (29). Due to potential statistical bias in FID’s reliance on Gaussian estimation, KID can be used alongside FID as a complementary non-parametric metric for evaluating generative image quality (30). The torch fidelity package was used to score the SHG image library as a baseline, the GAN generator, and the encoder with 10 stacks of 100 random images. To pass greyscale images to metric evaluation, the channel is repeated for each expected RGB layer. Additionally, StyleGAN2-ADA has built in FID metrics for in-training evaluation.

To further assess the baseline image database, generator, and encoder, structural similarity indexes (SSIM) were implemented for additional scoring. This index is computed locally across smaller windows of the image and then averaged, it evaluates intensity, contrast, and structure (31). Multiscale-SSIM is an extension that computes across multiple spatial resolutions, where it progressively downsamples the image to mimic the hierarchical nature of vision (32). MicroSSIM was developed in response to SSIM components behaving unexpectedly in images with low signal to noise conditions. Here it attempts to account for varying noise profiles and contrast levels common in microscopy datasets by incorporating background suppression and adaptive weighting strategies (33).

Fast Fourier transforms (FFTs) were used on all image databases to assess the exponential decay of the image with amplitude, offset, and the time constant in MatLab2023a. The FFT power spectrum captures the spatial frequencies contained in the image, where the fit provides information on the features in the tissues. A longer time constant will correspond to higher frequencies which in turn corresponds to smaller features in the collagen fibers (16).

3. Results

3.1. Metrics for overall training goals

To assess our StyleGAN2-ADA-SHG model, we analyzed the learned features and checked associations with collagen analysis tools for features of interest including collagen fiber length, width, curvature, density, and alignment. The training set includes widely varying examples of these features. Other differences in the samples include coverage (the overall signal of fibers across the 512 × 512 image), number of fibers, and brightness. To contextualize FID and KID in this heterogeneous SHG dataset, we computed a real–real split reference by comparing ten random pairs of 100-image subsets drawn from the training library (mean ± SD). The training data had respective FID 74.1 ± 1.44 and KID 7.41E-05 ± 6.54E-04 scores. Because Inception-based FID/KID rely on features trained on RGB natural images and FID assumes approximately Gaussian feature distributions, absolute values may be imperfect for grayscale SHG textures; we therefore interpret them cautiously and use them primarily as internal diagnostics (including monitoring training convergence) (34, 35).

3.2. StyleGAN2-ADA-SHG Training and Evaluation

After our model was trained, we first visually inspected generated images alongside real SHG images to assess quality. Specifically, we checked for repetitive artifacts, background noise, and fiber separation and fidelity (Figure 3). Additionally, the high variability between our SHG training data means that ideally, the generated images should also be similarly variable, encompassing a variety of collagen fiber densities, sizes, and textures. Our initial visual assessment was promising and suggested meaningful similarity between SHG and generated images (Figure 4). The FID-50k score from training was 25.7. Using torch-fidelity, the generator FID score was 92.2 ± 1.53 and the KID score was 0.00583 ± 0.00090 (mean ± SD over ten random evaluations of 100 generated vs. 100 real images). We report these alongside the real–real split reference (FID 74.1 ± 1.44) as dataset-specific diagnostics and interpret absolute Inception-based scores cautiously for SHG textures (36–38). Stability of training was assessed by inspecting the internal FID-50K curve (Figure 5), which indicated that 1,319 images were likely sufficient for stable training under this configuration. Full evaluation metrics are reported in Supplementary Table S.2; frequency-domain FFT diagnostics are summarized in Table S.3 and Figure S.3; and real versus synthetic feature distribution metrics are reported in Figure S.4.

Figure 3.

Grid of random samples from our StyleGAN2-ADA-SHG model. The generator must capture features found with collagen imaging like varying density, curvature, width, length, and alignment.

Figure 4.

Real SHG images (left) and randomly generated StyleGAN2-ADA-SHG images with similar texture (right). Generated images were inspected for similarity to a variety of imaged tissues.

3.3. Initial SeFa Analysis

SeFa computes the eigenvectors within StyleGAN that transform latent code to style code. These eigenvectors are then referred to as semantic directions that can traverse the latent space of the model. SeFa semantics for the generated images were analyzed with measures derived from CT-FIRE, CurveAlign, and FFT to look for semantics with significant changes to at least one feature. Two of the ten semantics showed correlation with one feature metric. All other features, including width, curvature, and mean density, did not show any significant changes. This analysis was performed with all layers (all style blocks) of the GAN (Table 1) on 10 random samples per semantic as an initial screening step for large effects. Effect size was calculated to examine the magnitude of difference between semantics weighted at +3 and −3. Semantic 001, correlated with length, was measured as having an effect size of 1.1, indicating a very large influence that is visually detectable. We note that the p-value of 0.048 for Semantic 001 does not survive Bonferroni correction across 10 semantics (α = 0.005); however, this initial screening was confirmed by the subsequent full traversal analysis using 1,050 observations, which yielded R² = 0.77 and PDC = 0.586 (see §3.4). Semantic 006 is included for reference as this analysis indicated no significant correlation with any feature metrics. Additional results are shown in Supplementary Table S.4 and Figures S.5–S.6.

Table 1.

Differences between selected semantics at weights of +3 and −3.

Semantic:	Semantic 001	Semantic 006
Mean Difference	Δ Length 2.08 ± 0.96	Δ Density 0.6 ± 1.6
Two-Tailed P-Value	0.048	0.71
Cohen’s D Effect Score	1.10	0.0289

3.4. Semantic Traversals on Identified Semantics

To further characterize the potential association of SeFa features with length and alignment, we traversed across semantic weights on all layers and all subsets of layers. Generally, in StyleGAN2-ADA, the style blocks of early levels correspond to coarse spatial resolution and control high-level features. Middle layers correspond to smaller scale features, and end layers correspond to higher resolution control such as pixel texture and microstructures.

In this network, Semantic 001 was most correlated with collagen fiber length. Results indicated that layer subset 3–4 best captured this feature with an R² value of 0.77 for a linear trendline across weight values (Figure 6). However, assessing the partial distance correlation scores (Table 2) indicates that layers 5–7 may have higher association with fiber length when controlling for the other variables. A higher score indicates a stronger relationship where one would mean a perfect deterministic dependence. These results indicate that collagen fiber length is captured by this network as a small-scale feature. Linear behavior is expected based on the factorization within SeFa. Certain morphological features are intrinsically coupled, and manipulating one inevitably affects the other. In this case, shrinking fiber length until fibers no longer register also reduces collagen density (Figure 7). These features co-vary due to shared spatial or structural constraints. Additional analysis of latent traversal coverage is available in Supplemental Table S.5.

Figure 6.

Results of CT-Fire analysis on collagen fiber length for Semantic 001 traversal applied to all layers, layers 0–2, layers 3–4, and layers 5–7. Linear trendlines and R² values are reported for each subset.

Table 2.

Partial distance correlation (PDC) of length assessed versus semantic weight, while controlling for fiber width, curvature, and density.

Semantic 001 – Length	PDC Score
Layers 0–2	0.203 ± 0.035
Layers 3–4	0.386 ± 0.031
Layers 5–7	0.586 ± 0.031

Figure 7.

Semantic 001 traversal results at layers 3–4 with weights −6, 0, and +6 from left to right. As length is increased and decreased, the overall structure of the image appears to stay the same. Individual fibers can also be seen shrinking and elongating.

Due to the trend observed in Semantic 001, we suspected that the initial SeFa analysis with all layers was insufficient for observing potential feature-semantic correlations. Therefore, we used our visualization module to explore all 10 semantics on all subsets of layers to observe possible trends. Visual inspection revealed that Semantic 006 showed a promising correlation with fiber density (defined, in this case, as the mean distance between nearest fibers). Layers 0–2 showed few differences. Layers 3–4 and 5–7 were then analyzed to pick the most representative and least entangled feature representation. Both layer subsets yielded similar slope magnitude (though one is inverted) and R² values for density (based on the mean distance to the nearest fibers). However, the response of the output density in Layers 3–4 appears to have entangled behavior with both fiber width and length. These measures yielded high R² values, confounding our expectation that a semantic would not be correlated with multiple features (Figure 8). Thus, Layers 5–7 are the best representation of fiber density for Semantic 006 (Table 3). The inverse slopes of these two-layer subsets could hypothetically be the cause of this semantic not producing significant results in the initial SeFa analysis of all layers combined (Table 1). If the behavior is of equal and opposite magnitude, the overall effect of the semantic may have gone undetected. This cancellation effect demonstrates why layer-subset analysis is essential and why the initial all-layers screening missed this semantic. Visual inspection of sample behavior appears to agree with this quantitative analysis, as Layers 3–4 appear to add more random texture than Layers 5–7. The images from Layer 5–7 appear to better maintain the original structure and arrangement (Figure 9). Additional analysis of latent traversal coverage is available in Supplemental Table S.6.

Figure 8.

Results of CT-FIRE analysis on collagen fiber length, width, straightness, and mean distance to nearest fibers (MDNF) to represent density. Linear trendlines and R² values are reported for each feature for both layer subsets.

Table 3.

Partial distance correlation (PDC) of density assessed versus semantic weight, while controlling for fiber width, curvature, and length.

Semantic 006 – Density	Partial Distance Correlation
Layers 0–2	0.303 ± 0.0430
Layers 3–4	0.384 ± 0.0443
Layers 5–7	0.657 ± 0.0405

Figure 9.

Semantic 006 traversal results at weights −6, 0, and +6 from left to right. Images from layers 3–4 are within the top row and appear to add more overall texture than Layers 5–7 (bottom row). However, this increased texture is more random and entangled with other features than in Layers 5–7.

3.5. Encoder Training and Projection

Our encoder was trained on GAN images and visually inspected for projection quality with both generated images and real SHG images (Figure 10). The encoder achieved an FID score of 68.1 ± 1.2 and a KID score of 0.00190 ± 0.00060. Stability of training was assessed by inspecting the KID curve. Noise in this measurement is due to hardware memory limitations on batch size, where smaller batches typically lead to greater variance in training (Figure 11). Visual inspection shows that projection does not completely capture the imported image, especially with real SHG images. This is expected, as encoders are typically liable to the loss of fine-grained detail (39). Analyses indicate that reconstruction errors are more pronounced for real SHG images than for synthetic inputs, particularly for fiber count, density, and organization. Additional details and full analysis of morphological metrics with real and synthetic encoded images are provided in Supplementary Figures S.7–S.8 and Table S.7.

Figure 10.

Images were mapped to the generator’s latent space using the encoder. To test the encoder’s fidelity both GAN images (top left) and real SHG images (middle and bottom left) were encoded.

Figure 11.

Encoder training was validated with Kernel Inception Distance score.

Results for the learned perceptual image patch similarity-based projection are shown in Supplementary Figure S.9.

4. Discussion

We have previously applied machine learning techniques to classify alterations in normal and diseased collagen fiber architectures in the ECM (4, 16, 40, 41). These efforts are relevant to human health as collagen fiber structures are altered in a wide range of pathologies, including cancers and fibroses. For example, the transition from normal ECM to dysplasia is often associated with the early stages of cancer. Further changes in the ECM structure are apparent in disease progression as in advanced tumor masses, the ECM can comprise up to 60% of the tissue, often dominated by desmoplasia or accumulation of newly synthesized collagen (42). While successful in terms of accuracy, our custom designed classifiers using textons and a 2D wavelet/PCA approach are still “black-box”. It is important to capture the specific features that are associated with diseases for more directed use. The ability to edit in latent space to determine the relative contributions of fiber characteristics, here semantics, adds further richness to the approach.

There are a few approaches to latent space analysis that may improve the number of meaningful semantics available. Alterations to images within latent space may produce image variations that are not visually detectable or captured within the current set of analysis tools. SeFa in particular typically corresponds to linearly separable variations that can affect the image independently. This separation ability is limited when there are higher-order interactions between latent components and may miss more complex structures in the latent space (43, 44). Alternative strategies could address these limitations. For example, GANSpace applies principal component analysis (PCA) to intermediate feature activations, which can capture combinations of latent factors (45). We attempted GANSpace with our StyleGAN2-ADA-SHG model, but the available implementation was not compatible with our training environment (dependency/CUDA stack conflicts), preventing the batch-based pipeline needed for systematic statistical evaluation. Future approaches could adapt the methodology to modern toolkits. StyleFlow uses normalizing flows, which are explicitly nonlinear for modeling complex semantic trajectories (46) and could be another useful strategy for finding features such as curvature and width that are not captured by SeFa’s linear factorization.

For our apparent fiber-based features, we built the semantic traversal tool to capture the behavior of the eigenvectors given randomly generated images. In general, we expect to correlate semantics to collagen fiber features in the CT-Fire and FIJI analysis including length, width, curvature, density, and alignment. As a neural network trains on a dataset, it is expected to capture hierarchical features of images including edges, shapes, and textures as a natural result of successive convolutional and downsampling layers. However, features are also expected to be connected to the layers of Style blocks within the network. In this study, when semantics with changes were detected in the initial analysis, they were later found to have more linear behavior in specific subsets of layers. Specifically, we found two semantics correlated with collagen fiber length and density respectively.

The baseline data, generator, and encoder all had relatively high FID scores and relatively low SSIM scores. There are a few potential confounding factors that can influence these results in the SHG imaging domain. Features used by FID and KID are derived from ImageNet-trained models on labeled RGB datasets and may have limited value for grayscale microscopy textures (47). FID values can also change with sample size (48), and SSIM can be lower when signal-to-noise ratio varies across a heterogeneous dataset (49). In our case, the use of multiple tissue types and disease states introduces substantial heterogeneity in the sample library. To contextualize this domain, we performed a targeted literature search and did not find prior reports of FID for generative models trained on SHG microscopy images. Instead we complement FID/KID/SSIM with collagen-relevant validation using CT-FIRE/CurveAlign fiber metrics, FFT frequency diagnostics, and partial distance correlation (PDC) analyses for the robust semantic directions. Classifier-based “fooling” evaluations are beyond the scope of this study, but would be best implemented into our pipeline for further characterization using stratified training methods in larger datasets for the GAN.

There are several benefits in using a library of SHG images from multiple tissue and disease types. In general, increasing the number of images in training significantly increases the quality of the resulting generator network. Specific to microscopy imaging of human data, the largest issue for achieving high quality machine learning is the small number of images available for training. This, by combining images of different pathologies, we have significantly decreased the probability of low-data biases in training. Due to these factors, disease labels were intentionally not incorporated in this initial model in order to maximize available training data and enable the generator to learn general collagen morphology features across the full scope of tissue types. However, this introduces a limitation to model interpretation: because the training set pooled multiple tissue types without disease labels, the two identified semantics (fiber length and density) could hypothetically reflect inter-tissue morphological variation rather than disease-specific collagen alterations. Identifying disease specific features will require stratified (e.g. cancer stage) training and validation in future work. Moreover, a larger data set could also reveal additional significant semantics than found here. The dataset composition, including the number of images per tissue category and available imaging parameters, is documented in Supplementary Tables S.1.

In general, converting neural networks to work with greyscale images reduces the dimensionality and therefore the overall complexity of a given network. It also slightly reduces the number of learnable parameters and computational load compared to a network with RGB input. In principle, this also simplifies feature extraction across the model. However, a major trade-off is the inability to effectively use common pre-trained datasets such as ImageNet for transfer learning. Additionally, FID and KID scores are based on a model trained on RGB image features. By triplicating the greyscale input, we can utilize these metrics, but there may be unpredictability in evaluating scores and a less stable correlation to visual inspection.

In this study, we have adapted a StyleGAN2 encoder architecture for use in projecting images into the model’s network. Our reconstruction fidelity for real SHG images was assessed qualitatively based on visual preservation of gross fiber architecture and directionality. Supplementary Table S.7 shows that the encoder introduces statistically significant deviations in fiber count, width, and mean nearest-neighbor distance for real inputs, while synthetic image reconstruction preserves most metrics without significant differences. In future work, formal quantitative acceptance criteria for real-image editing should be established alongside improvements to overall encoder performance.

Explainable machine learning can lead to further study of biological features in disease states. An ideal model is transparent, interpretable, and explainable (50). Given the rapid pace of innovation in machine learning, it is important to consider alternative generative models to StyleGAN2-ADA that can offer similar capabilities. StyleGAN3 was created in part to address phase artifacts that can occur due to the discriminator’s progressive architecture. However, there are substantial hurdles that must be overcome to convert this method to use a greyscale-only network (51). CycleGAN is widely used for unpaired microscopy image translation tasks such as virtual staining and modality transfer (52–54). However, it is fundamentally a domain translation model rather than a de novo generator, requiring two reference image domains and producing deterministic one-to-one mappings. Its encoder-decoder architecture also yields a 4D intermediate feature tensor rather than the 2D weight matrix that SeFa’s eigen decomposition requires, making direct latent factorization incompatible without substantial architectural modification.

Variational autoencoders (VAE) aim to model the underlying distribution of data through a compressed latent space. This method uses a Gaussian estimation to approximate the posterior distribution of the data. Although this creates a relatively smooth, continuous, and interpretable latent space, the Gaussian assumption can result in blurry image generation (55, 56). There are methods that address these limitations, but they do not offer the same degree of ease in editing the latent codes. For example, the VQ-VAE model provides high-quality image generation while using discrete hierarchical latent codes rather than those who are continuous (57). NVAE uses convolutions within the generator to improve results and separates the latent space codes with deep hierarchical architecture (58). However, both approaches require a separate module to render images in response to latent codes being accessed and edited.

Latent Diffusion Models (LDMs) extend upon the VAE framework by building a hierarchy of denoising autoencoders that operate in the lower-dimensional latent space rather than the standard pixel space. They have been particularly adept at modeling complicated multi-modal distributions (59). The approaches to editing latent space features in latent diffusion models have generally focused on text-to-image generation. However, recent approaches have focused on linear and versatile image-editing that more closely resembles the relatively straightforward process of editing the latent space within GANs (60, 61). Despite advances in data efficiency, diffusion models often require substantially more data and/or careful conditioning to achieve high fidelity in specialized microscopy domains (62). In contrast, StyleGAN2-ADA-SHG could be trained effectively on 1,319 images in our setting.

5. Conclusions

In this study, we implemented a high-performing GAN, StyleGAN2-ADA-SHG, for creating SHG images of collagen fibers, alongside GUI tools which allow for fine control of image manipulation that aid in the analysis of the learned semantics within latent space. Using external analysis tools, CT-Fire and FIJI, we directly modeled the GAN response within latent space. By testing each subset of layers using SeFa-based semantic sliders, we were able to directly correlate features to their respective network style block within StyleGAN2-ADA-SHG. This quantitative approach can greatly aid the usage of GANs and potentially allow for new ways to analyze morphological changes in collagen fibers in disease states, even in small data sets. Ultimately, although only two meaningful semantics were discovered herein, this work demonstrates that latent space analysis of SHG microscopy images is a viable strategy for identifying and manipulating collagen fiber morphological features. The pipeline established here, combining StyleGAN2-ADA-SHG, SeFa semantic factorization, and quantitative fiber analysis, provides an extensible framework that can be built upon with additional semantics, nonlinear latent space methods, disease-stratified training, and improved encoder architectures in future studies.