Image-based analysis of lymphoma cell heterogeneity: Comparative study of CRYO and FFPE techniques

Abstract

Lymphomas comprise a diverse group of heterogeneous cell malignancies, primarily diagnosed through morphological and histochemical analysis of tissue biopsy specimens. Accurate automated nuclear segmentation is essential for quantitative histopathological analysis and downstream computational pathology applications, particularly in morphological heterogeneous lymphoma tissues. In the current study, we used deep learning-based approaches, namely a pre-trained StarDist model, a custom-trained StarDist model, and DeepLabV3, for automated nuclear segmentation of heterogeneous lymphoma cells in histological sections and achieved high accuracy of nuclear segmentation. DeepLabV3 achieved the highest pixel-level segmentation accuracy when trained on large annotated datasets, whereas StarDist demonstrated greater data efficiency under limited training conditions. We next compared the accuracy of nuclear segmentation between conventional formalin-fixed paraffin-embedded (FFPE) specimens and recently proposed glyoxal-based processing of rapidly frozen samples (CRYO). Overall, segmentation performance on CRYO specimens was comparable to that on FFPE specimens, although the magnitude and direction of differences varied by models. Custom-trained StarDist achieved a modestly higher Dice coefficient on CRYO specimens, whereas DeepLabV3 showed only a small, non-significant difference, and the pre-trained StarDist model showed no significant difference between tissue preparations. DeepLabV3 provided superior pixel-level segmentation accuracy, whereas StarDist demonstrated more accurate boundary localization, instance-level segmentation, and greater robustness to reduced training data. We conclude that both StarDist and DeepLabV3 can provide high-precision nuclear segmentation across FFPE and CRYO lymphoma specimens, supporting the applicability of CRYO processing for computational pathology workflows.

Highlights

• •StarDist and DeepLabV3 enable accurate lymphoma cell segmentation.
• •DeepLabV3 shows higher performance than StarDist with large datasets.
• •StarDist is effective for limited training data.
• •CRYO glyoxal fixation produces FFPE-comparable histology quality.
• •CRYO and FFPE sections support high-precision segmentation.

Introduction

Accurate nuclear segmentation in histopathological images is a fundamental step for quantitative analysis in computational pathology.¹ Manual annotation of cells or nuclei is time-consuming and subject to inter-observer variability,² motivating the development of automated approaches that improve efficiency and reproducibility.³ Traditional segmentation methods, including thresholding, watershed, and region-growing techniques,4, 5, 6 are often limited by sensitivity to staining variability, image noise, and the presence of overlapping cells, restricting their applicability in complex tissue environments.

Hematoxylin-stained sections of lymphoid tissue present a particularly challenging setting for automated nuclear segmentation due to high cellular density, morphological heterogeneity, and low contrast between adjacent nuclei. These characteristics are especially pronounced in lymphoma-affected tissues, where densely packed cells and diverse nuclear morphologies complicate accurate delineation.⁷ As a result, such tissues provide a rigorous benchmark for evaluating segmentation methods in histopathology.

Recent advances in deep learning (DL), particularly convolutional neural networks (CNNs)-based approaches, have significantly improved automated nuclear segmentation performance across biomedical imaging applications.2, 8, 9, 10, 11 However, most studies have focused on images obtained by fluorescence microscopy or relatively simple imaging context, where contrast is high and cellular structures are more easily separable. In contrast, segmentation of nuclei in histological tissue sections using brightfield microscopy remains challenging due to complex tissue architecture and variability in specimen preparation.7, 12, 13 These factors highlight the importance of evaluating segmentation methods under realistic histopathological conditions and across different tissue-processing protocols.

Among these factors, tissue fixation and processing can alter cellular morphology and tissue characteristics and may consequently affect downstream image analysis.¹⁴ Formalin-fixed paraffin-embedded (FFPE) processing remains the gold-standard in histopathology. Recently, glyoxal-based fixation of rapidly frozen samples (CRYO) has been introduced as a rapid alternative.¹⁵ Glyoxal fixation has been shown to preserve cellular morphology and staining properties comparable to formalin, whereas offering advantages such as improved structural preservation and reduced processing time.16, 17 Despite these benefits, the suitability of glyoxal-based CRYO processing for DL-based segmentation of histological tissue, particularly in comparison with FFPE processing, has not been systematically evaluated.

In this study, we investigated the performance of two established DL architectures, StarDist,¹⁸ an instance-segmentation approach designed to delineate individual star-convex objects, and DeepLabV3,¹⁹ a semantic-segmentation architecture based on atrous convolution and multi-scale feature extraction. Both approaches were evaluated for automated nuclear segmentation of hematoxylin-stained lymphoid tissue prepared using conventional FFPE and glyoxal-based CRYO protocols. This design enabled assessment of whether differences in tissue preparation were associated with segmentation performance across complementary DL approaches.

The objectives of this study were to: (1) quantitatively assess nuclear segmentation performance in hematoxylin-stained lymphoid tissue using StarDist and DeepLabV3, and (2) evaluate whether glyoxal-based CRYO preparation provides segmentation performance comparable to conventional FFPE processing. By evaluating both specimen preparation protocols using patient-level validation and independent held-out data, we aimed to assess the feasibility of CRYO-prepared specimens for DL-based quantitative histopathological analysis. Our findings indicate that CRYO-prepared specimens achieved segmentation performance comparable to FFPE-prepared specimens in our datasets, supporting the potential applicability of glyoxal-based CRYO processing in computational pathology workflows.

Materials and methods

Tissue samples and study design

Lymph node biopsy specimens from patients with suspected lymphoma were obtained from the National Scientific Center for Oncology and Transplantation (Astana, Kazakhstan). Ethical approval was granted by the institutional review board (Approval No. 9, June 21, 2021), and all procedures were conducted in accordance with relevant guidelines and regulations. Written informed consent was obtained from all patients.

The cohort included 18 lymph node specimens, including 10 Hodgkin lymphoma cases (classical subtype), 7 non-Hodgkin lymphoma cases (2 marginal zone lymphoma, 2 follicular lymphoma, 1 mantle cell lymphoma, 1 chronic lymphocytic leukemia/small lymphocytic lymphoma (CLL/SLL), and 1 large B-cell non-Hodgkin lymphoma not further identified), and 1 lymph node specimen with cancer metastasis. Immediately after excision, tissue samples were placed in RPMI 1640 medium and transported to the lab within 1 h. Each specimen was divided into two portions for parallel histological processing using FFPE and cryosectioning with glyoxal fixation (CRYO) protocols.

Histological processing

FFPE preparation

Tissue samples were fixed in 4% (w/v) paraformaldehyde for 24 h, followed by washing in phosphate-buffered saline. The samples were dehydrated through graded ethanol solutions, cleared in xylene, and embedded in paraffin. Sections of 4 μm thickness were cut, mounted on glass slides, deparaffinized, and rehydrated using standard protocols.²⁰

CRYO preparations

Parallel tissue fragments (2–3 mm thickness) were snap-frozen in n-hexane cooled with liquid nitrogen and sectioned using a cryostat (≤7 μm thickness). Sections were fixed in 3% (v/v) glyoxal solution for 8 min at room temperature, rinsed in running water, and air-dried before staining. Glyoxal fixative solution was prepared as previously described.¹⁵

Hematoxylin staining and image acquisition

Both FFPE and CRYO sections were stained with hematoxylin, dehydrated, and mounted using standard protocol. Images were acquired using a Leica DM300 microscope equipped with a Leica ICC50 HD camera using ×40/0.75 PlanFluor objective, corresponding to a spatial resolution 0.16 μm/pixel. At least three non-overlapping fields of view were captured per slide.

Annotation and ground-truth generation

Manual nuclear annotations were performed using QuPath software.²¹ Images were divided into smaller patches before manual annotation to increase the diversity of training samples. Each image was annotated at the pixel level, with individual nuclei assigned unique labels and the background assigned to zero.

To assess inter-observer variability, a subset of 50 images (25 FFPE and 25 CRYO) was independently annotated by 2 trained histologists. Inter-observer agreement was assessed using the Dice coefficient, yielding values of 0.8383 ± 0.0668 for FFPE and 0.8371 ± 0.0529 for CRYO images, with an overall Dice coefficient of 0.8377 ± 0.0596. All annotations underwent quality-control review before inclusion in the final dataset.

The final ground-truth dataset comprised 84,744 manually delineated nuclei across 464 images, corresponding to a mean of 183 nuclei per image.

Dataset comparison and splitting strategy

The annotated dataset comprised 231 FFPE and 233 CRYO images obtained from 18 patients. The 84,744 annotated nuclei comprised 45,779 nuclei in FFPE images and 38,965 in CRYO images. The mean number of annotated nuclei was 198 ± 68 per FFPE image (median, 195; range, 67–463) and 167 ± 56 per CRYO image (median 160; range 46–303), yielding an overall mean of 183 ± 64 nuclei per image. The median nuclear area was 745 pixels, corresponding to an equivalent circular diameter of 30.8 pixels (4.93 μm) at a spatial resolution of 0.16 μm/pixel.

To prevent data leakage, all data splits were performed at the patient level, such that images from a given patient was assigned exclusively to either training, validation, or evaluation set. FFPE and CRYO images were analyzed separately. The pre-trained StarDist model was applied without any training or fine-tuning on the study dataset; therefore, cross-validation was not required, and all images were evaluated in a single inference pass. For the custom-trained StarDist model and DeepLabV3, evaluation was performed using a leave-one-patient-out (LOPO) procedure. In each iteration, one patient was held out for evaluation, whereas the remaining patients were split using 5-fold cross-validation into training and validation sets. A model was trained from scratch for each fold and evaluated on the held-out patient. The reported value for each patient represents the mean Dice score across the five folds, whereas the final row reports the mean ± SD across patient. Thus, each patient was evaluated exclusively by models trained without images from that patient.

Deep learning models

StarDist

We evaluated both a pre-trained StarDist model (2D_versatile_he), specifically designed for nuclei segmentation in brightfield H&E-stained images, and a custom-trained version adapted to our dataset.¹⁸ StarDist performs instance segmentation by predicting object probabilities and radial distances to object boundaries, followed by non-maximum suppression (NMS) to resolve overlapping predictions.

For the pre-trained 2D_versatile_he model, RGB brightfield images were used without intensity inversion or stain deconvolution. Joint percentile normalization was applied across the RGB channels before inference, and a NMS threshold of 0.4 was used.

The custom StarDist model was trained for up to 300 epochs with 100 steps per epoch. Validation Dice coefficient was monitored for model selection, and early stopping with a patient of 10 epochs was applied. The model checkpoint achieving the highest validation Dice coefficient was retained for evaluation. RGB input images were normalized across channels. Probability and NMS thresholds were empirically optimized using the training/validation data and fixed before evaluation on the held-out patients.

DeepLabV3

DeepLabV3¹⁹ was implemented for semantic segmentation. The architecture incorporates atrous convolution and atrous spatial pyramid pooling (ASPP) to capture features at multiple spatial scales. An ImageNet-pretrained EfficientNet-B7 encoder was used, implemented through the segmentation-models-pytorch library. The encoder contained 63.8 million parameters, and the complete model contained 69.5 million parameters. No encoder layers were frozen, and the model was fine-tuned end to end with an output stride of 16 (Fig. 1).

Fig. 1.

Architecture of the DeepLabV3 model.

Input images were resized to 512 × 512 pixels and normalized to [0,1]. Contrast-limited adaptive histogram equalization was applied to the luminance channel of the CIE-Lab representation using a clip limit of 2.0 and an 8 × 8 tile grid. Data augmentation included random horizontal and vertical flips, rotations of ±15°, random shifts of ±5%, random scaling of ±10%, and elastic deformation with a probability of 0.3.

The model was trained for 100 epochs using the Adam optimizer with an adaptive learning rate schedule. Dice loss was used as the optimizer objective. The initial learning rate was 3 × 10⁻⁴ and was reduced by a factor of 0.5 after 8 consecutive epochs without improvement in validation Dice coefficient. Mixed-precision training was used with an effective batch size of 8.

Instance extraction from DeepLabV3 predictions

Because DeepLabV3 produces a semantic foreground probability map rather than individual nuclear instances, an additional post-processing procedure was applied before instance-level evaluation. The predicted probability map was thresholded at 0.5 to generate a binary foreground mask. An Euclidean distance transform was then computed and smoothed using a Gaussian filter (sigma = 1.0 pixels).

Local maxima of the smoothed distance map were identified as seed points, restricted to the foreground region and separated by a minimum peak distance of six pixels. Each seed was assigned a unique marker label, and marker-controlled watershed segmentation was subsequently applied to the negated distance transform within the foreground mask. No compactness constraint or morphological opening/closing was applied.

Objects smaller than 30 pixels (0.77 μm² at 0.16 μm/pixel) were removed as debris after watershed segmentation, and the remaining instances were renumbered consecutively. The debris threshold used was substantially below the median nuclear area of 745 pixels. Across the annotated dataset, only 3 of 79,933 components contained fewer than 20 pixels, indicating that this filtering step had minimal effect on nuclear instances.

Watershed labels were retained directly and were not subjected to subsequent connected-component relabeling, which could merge adjacent instances sharing a common boundary.

All instance-level metrics for DeepLabV3, including F1 score, AJI, and PQ, were calculated from the resulting instance masks rather than from the binary semantic masks. The ground-truth masks were based exclusively on manual nuclear annotations and were not generated using the prediction post-processing procedure.

The implementation used scikit-image version 0.26.0 and SciPy version 1.17.0.

Training environment

All models were trained on a CUDA-enabled workstation (Ubuntu 22.04) equipped with an NVIDIA RTX 6000 Ada GPU. The software environment included Python 3.11 and PyTorch.

Inference time was measured on the NVIDIA RTX 6000 Ada GPU. For 512 × 512-pixel inputs, the DeepLabV3 forward-pass time was 35 ms per image, whereas StarDist required 80 ms per image including CPU-based NMS. These measurements are not directly comparable because the StarDist values includes instance-level post-processing, whereas the initial DeepLabV3 measurement represents only the neural-network forward pass.

For an end-to-end comparison, DeepLabV3 inference was additionally evaluated together with the watershed instance-extraction procedure described in Section Instance extraction from DeepLabV3 predictions. In a reference implementation, the DeepLabV3 forward pass required 16.7 ± 0.2 ms and watershed-based instance extraction required 84.3 ± 21.0 ms, resulting in an end-to-end processing time of 101.0 ms per image. Accordingly, inference times should be compared on an equivalent end-to-end basis rather than using forward-pass time alone.

Evaluation metrics

Segmentation performance was evaluated using the Dice coefficient, F1 score, Jaccard index (intersection over union; IoU), aggregated Jaccard index (AJI), 95th-percentile Hausdorff distance (HD95), and panoptic quality (PQ).22, 23, 24, 25 These metrics quantify complementary aspects of segmentation performance, including pixel-level overlap, instance-level detection and segmentation accuracy, and boundary precision.

Dice coefficient and IoU, instance-level F1 score, AJI, and PQ were calculated from binary foreground masks. HD95 was calculated for matched predicted and ground-truth nuclear instances as the maximum of the two directed 95th-percentile distances between their boundary pixel sets of then averaged across instances. The 95th percentile was used instead of the maximum to reduce sensitivity to isolated boundary outliers.

Statistical analysis

Statistical analysis was performed on patient-level values. Comparisons between FFPE and CRYO sections were treated as paired comparisons for all patients. Comparisons between StarDist and DeepLabV3 were likewise performed using paired patient-level observations for the common evaluation cohort.

Normality of paired differences was assessed using the Shapiro–Wilk test. When normality was not rejected (p ≥ 0.05), the paired t-test was used as the primary analysis; otherwise, the Wilcoxon signed-rank test was used.

For each comparison, the mean paired difference with its 95% confidence interval and Cohen's d_z effect size were reported. Where multiple related outcomes were tested, p values were adjusted using the Benjamini–Hochberg procedure to control the false-discovery rate. Statistical significance was defined as an adjusted p < 0.05 for multiplicity-corrected analysis and p < 0.05 otherwise.

Unpaired tests were not used for FFPE-versus-CRYO comparisons because the measurements were obtained from matched patients.

Results

Performance of pre-trained StarDist model

The performance of the pre-trained StarDist model was evaluated on FFPE and CRYO lymphoid images from 18 patients (Table 1). The mean Dice coefficient was 0.7432 ± 0.0374 for FFPE and 0.7398 ± 0.0315 for CRYO images (Table 1). The paired difference between CRYO and FFPE was −0.0034 (95% CI: −0.0254 to 0.0186; paired t-test, t(17) = −0.33, p = 0.75; Wilcoxon signed-rank test, p = 0.60; Cohen's d_z = −0.08).

Table 1.

Per-patient Dice coefficients for the pre-trained StarDist model (2D_versatile_he), applied without training on the study dataset. Each value is the mean over that patient's images. The final row reports the mean ± standard deviation (SD) across patients.

Patient	FFPE Dice	CRYO Dice	∆ (CRYO-FFPE)
L04	0.7430	0.7788	+0.0358
L07	0.7034	0.7991	+0.0957
L08	0.7714	0.7628	−0.0086
L10	0.7638	0.8083	+0.0445
L23	0.6296	0.6967	+0.0671
L24	0.7247	0.7570	+0.0324
L27	0.7827	0.7341	−0.0486
L39	0.7490	0.7379	−0.0111
L60	0.7108	0.7270	+0.0161
L61	0.7196	0.7307	+0.0112
L62	0.7468	0.7149	−0.0319
L63	0.7544	0.7029	−0.0514
L64	0.7562	0.7143	−0.0419
L65	0.7420	0.7192	−0.0227
L66	0.7759	0.7135	−0.0624
L68	0.7438	0.7239	−0.0200
L69	0.7670	0.7415	−0.0255
L70	0.7934	0.7533	−0.0401
Mean ± SD	0.7432 ± 0.0374	0.7398 ± 0.0315	−0.0034 ± 0.0442

Thus, the pre-trained model showed comparable pixel-level segmentation performance across the two specimen preparation methods without dataset-specific fine-tuning. Although individual patient-level differences were observed, no consistent advantage of either preparation method was evident.

Performance of custom-trained StarDist model

The custom-trained StarDist model was evaluated by LOPO: each patient was held out in turn, models were trained on the remaining patients using 5-fold cross-validation, and the patient's score is the mean across 5 folds (Table 2).

Table 2.

Per-patient Dice coefficients for the custom-trained StarDist model. Each patient was held out in turn and scored by models trained without that patient's images. The tabulated value is mean across the five inner folds. The final row reports the mean ± SD across patients.

Patient	FFPE Dice	CRYO Dice	∆ (CRYO-FFPE)
L04	0.7488	0.7792	+0.0303
L07	0.7098	0.7727	+0.0629
L08	0.7637	0.7906	+0.0269
L10	0.7645	0.8014	+0.0368
L23	0.6431	0.7713	+0.1283
L24	0.7402	0.7896	+0.0494
L27	0.7908	0.7899	−0.0009
L39	0.7531	0.7864	+0.0333
L60	0.8136	0.7776	−0.0360
L61	0.7371	0.7887	+0.0516
L62	0.7574	0.7759	+0.0185
L63	0.6290	0.6742	+0.0453
L64	0.7559	0.7866	+0.0307
L65	0.6585	0.7496	+0.0911
L66	0.7799	0.7881	+0.0083
L68	0.7184	0.7372	+0.0188
L69	0.7632	0.7125	−0.0506
L70	0.7093	0.6028	−0.1065
Mean ± SD	0.7354 ± 0.0499	0.7597 ± 0.0506	+0.0243

Mean Dice coefficients were 0.7354 ± 0.0499 for FFPE and 0.7597 ± 0.0506 for CRYO images. CRYO yielded higher Dice coefficient than FFPE, with a mean paired difference of 0.0243 (95% CI, −0.0016–0.0503; paired t-test, t(17) = 1.98, p = 0.0641; Wilcoxon signed-rank test, p = 0.0483; Cohen's d_z = 0.47).

Compared with the pre-trained model, custom training produced only a small decrease for FFPE images (mean decrease, −0.0078; paired t-test, p = 0.51; Wilcoxon signed-rank test, p = 0.67), whereas for CRYO images, we observed an improvement (mean increase, 0.0199; paired t-test, p = 0.15; Wilcoxon signed-rank test, p = 0.035).

Effect of training-set size on StarDist performance

The sensitivity of StarDist to training-set size was assessed using 10 repeated patient-level splits with training proportions of 80%, 60%, 40%, and 20%. For the 18-patient cohort, these proportions corresponded to 14, 11, 7, and 4 patients in the training set, respectively.

StarDist performance remained relatively stable as the training set decreased (Fig. 2, Fig. 3). CRYO images showed particularly limited variation across training proportions, whereas FFPE images showed a modest decline in Dice coefficient with decreasing training-set size. Overall, the model maintained relatively stable performance even under reduced-data conditions. However, the 20% training condition included only approximately four patients and should therefore be interpreted as a descriptive assessment of data sensitivity within the present cohort rather than an evidence of generalizable data efficiency.

Fig. 2.

CRYO specimen performance across train-test ratios for StarDist. Box plots show Dice coefficient distributions obtained using 10 repeated patient-level splits. Orange lines indicate mean values; circles represent outliers.

Fig. 3.

FFPE specimen performance across train-test ratios for StarDist. Box plots show Dice coefficient distributions obtained using 10 repeated patient-level splits. Orange lines indicate mean values; circles represent outliers.

Performance of DeepLabV3 model

DeepLabV3 was evaluated using the same 18-patient cohort and patient-level cross-validation (Table 3).

Table 3.

Per-patient Dice coefficients for DeepLabV3 model. Each patient was held out in turn and scored by models trained without that patient's images. The tabulated value is the mean across the five inner folds. The final row reports the mean ± SD across patients.

Patient	FFPE Dice	CRYO Dice	∆ (CRYO-FFPE)
L04	0.8530	0.8736	+0.0206
L07	0.8616	0.8133	−0.0483
L08	0.8486	0.7805	−0.0681
L10	0.8585	0.8349	−0.0236
L23	0.8508	0.7844	−0.0664
L24	0.8652	0.8436	−0.0216
L27	0.8685	0.7704	−0.0981
L39	0.8624	0.8741	+0.0117
L60	0.7753	0.8475	+0.0722
L61	0.8557	0.8518	−0.0039
L62	0.7984	0.8061	+0.0077
L63	0.8668	0.8702	+0.0034
L64	0.8629	0.8145	−0.0484
L65	0.7805	0.8454	+0.0649
L66	0.7953	0.7853	−0.0100
L68	0.7575	0.7943	+0.0368
L69	0.8523	0.8343	−0.0180
L70	0.8587	0.8054	−0.0533
Mean ± SD	0.8373 ± 0.0370	0.8239 ± 0.0335	−0.015

Mean Dice coefficients were 0.8373 ± 0.0370 for FFPE and 0.8239 ± 0.0335 for CRYO specimens. The difference between fixation methods was not statistically significant paired t-test comparing patient-level Dice coefficients showed no statistically significant difference between the FFPE and CRYO datasets (mean paired difference, −0.015; paired t-test, t(17) = −1.32, p = 0.21).

Training convergence was assessed to evaluate model stability and potential overfitting. Training and validation Dice coefficients increased concurrently during the initial 25 epochs, after which the curves gradually diverged. Validation Dice plateaued at approximately 0.74, whereas training Dice increased modestly to 0.79, resulting in a final difference of 0.049. The highest validation Dice (0.7452) was achieved at epoch 181, compared with 0.7446 at epoch 100, indicating that extending training to 200 epochs provided negligible improvement (<0.0001). These findings suggest mild overfitting without evidence of substantial training instability (Supplementary, Fig. 1).

Compared with custom-trained StarDist, DeepLabV3 achieved higher pixel-level segmentation accuracy for both FFPE and CRYO images.

Effect of training-set size on DeepLabV3 performance

DeepLabV3 was evaluated using the same 4 training proportions and 10 repeated patient-level splits used for StarDist (Fig. 4, Fig. 5). In contrast to StarDist, DeepLabV3 performance decreased substantially as the training-set size was reduced.

Fig. 4.

CRYO specimen performance across train-test ratios for DeepLabV3. Box plots show Dice coefficient distributions obtained using 10 repeated patient-level splits. Orange lines indicate mean values; circles represent outliers.

Fig. 5.

FFPE specimen performance across train-test ratios for DeepLabV3. Box plots show Dice coefficient distributions obtained using 10 repeated patient-level splits. Orange lines indicate mean values; circles represent outliers.

For FFPE images, mean Dice coefficients decreased from approximately 0.82 at 80% training data to 0.34 at 20% training data. For CRYO images, the corresponding values decreased from approximately 0.83 to 0.36. The reduced-data conditions also showed substantially greater variability than the full-data condition.

These findings indicate greater sensitivity of DeepLabV3 than StarDist to training-set size. Because the 20% condition involved only four training patients, the magnitude of the observed decrease should be interpreted cautiously.

Comparative performance of StarDist and DeepLabV3

The comparative performance of the two architectures is summarized in Table 4.

Table 4.

Comparative segmentation performance across FFPE and CRYO tissues. Dice: region overlap; Jaccard (IoU): intersection over union; AJI: Aggregated Jaccard Index; HD95: 95th-percentile Hausdorff distance in pixels (lower is better); PQ: Panoptic quality. Values are reported as mean ± standard deviation.

Model	Specimen	Dice	Jaccard	AJI	HD95 (px)	PQ
StarDist	FFPE	0.736 (50)	0.62 (4)	0.570 (4)	5.25 (110)	0.576 (4)
	CRYO	0.760 (50)	0.64 (4)	0.583 (4)	6.65 (95)	0.613 (3)
DeepLabV3	FFPE	0.836 (38)	0.70 (5)	0.630 (5)	8.70 (125)	0.560 (4)
	CRYO	0.821 (32)	0.69 (4)	0.620 (4)	8.50 (115)	0.550 (4)

Note: Numbers in parentheses denote the standard deviation in units of the last displayed digit (e.g., 0.754 (41) = 0.754 ± 0.041; 8.70 (125) = 8.70 ± 1.25).

DeepLabV3 achieved higher Dice and Jaccard indices than custom-trained StarDist for both fixation methods. At the patient level, the mean Dice coefficient was higher for DeepLabV3 than StarDist by 0.1020 for FFPE (95% CI, 0.0697–0.1343, paired t-test, t(17) = 6.67, p < 0.0001; Cohen's d_z = 1.57) and by 0.0642 for CRYO (95% CI, 0.0332–0.0952, paired t-test, t(17) = 4.37, p = 0.0004; Cohen's d_z = 1.03).

The Wilcoxon signed-rank test yielded concordant results for FFPE (p < 0.0001) and CRYO (p = 0.0013). After Benjamini–Hochberg correction for the pre-specified set of patient-level Dice comparisons, the difference between DeepLabV3 and StarDist remained statistically significant for both FFPE and CRYO.

Despite its higher region-overlap accuracy, DeepLabV3 produced higher HD95 values than StarDist, indicating less precise boundary localization. StarDist showed lower HD95 values for both FFPE and CRYO images (5.25 and 6.65 pixels, respectively) than DeepLabV3 (8.50–8.70 pixels, respectively).

StarDist also achieved higher PQ values than DeepLabV3 (0.576 vs. 0.560 for FFPE and 0.613 vs. 0.550 for CRYO), indicating better instance-level segmentation performance under the panoptic-quality criterion. Conversely, DeepLabV3 achieved higher AJI values, indicating better overall agreement of aggregated instance regions.

Overall, DeepLabV3 achieved substantially higher pixel-level overlap than StarDist, whereas StarDist demonstrated lower HD95 values, higher PQ, and greater robustness to reductions of training-set size. These findings indicate complementary strengths of the two approaches: DeepLabV3 provided superior foreground segmentation, whereas StarDist showed advantages in boundary localization, instance-level segmentation, and performance under limited training data. These differences are also evident in representative qualitative segmentation results for CRYO and FFPE specimens (Fig. 6).

Fig. 6.

Qualitative comparison of nuclear segmentation results in CRYO and FFPE specimens. The upper row shows four representative CRYO specimens, whereas the lower row shows four representative FFPE specimens. Purple indicates ground-truth nuclear annotations, blue indicates StarDist predictions, and green indicates DeepLabV3 predictions following watershed-based instance extraction. Scale bars, 10 μm. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Data efficiency analysis

Model sensitivity to training-set size is summarized in Table 5.

Table 5.

Data efficiency comparison. Performance retention is calculated as the ratio of Dice at 20% training data to Dice at 80% training data.

Model	Specimen	Dice (80%)	Dice (20%)	∆	Retention
StarDist	FFPE	0.77	0.73	−0.04	95%
	CRYO	0.78	0.73	−0.05	94%
DeepLabV3	FFPE	0.82	0.34	−0.48	41%
	CRYO	0.83	0.36	−0.47	43%

Relative to the 80% training condition, StarDist retained approximately 94%–95% of its Dice performance when trained with only 20% of the available patients. In contrast, DeepLabV3 retained approximately 41%–43% of its full-data performance.

The greater reduction in DeepLabV3 performance under limited-data conditions was accompanied by increased variability across repeated patient-level splits. These findings indicate that, within the present cohort, StarDist was less sensitive to reductions in training-set size than DeepLabV3.

Because the 20% condition corresponded to only four training patients, these results should be interpreted as cohort-specific evidence of relative data sensitivity rather than as a general measure of architectural data efficiency.

Statistical comparison across specimen preparation methods

The effect of tissue-preservation method on segmentation performance was assessed by paired comparisons of FFPE and CRYO Dice coefficients at the patient-level for each of the three model configurations: pre-trained StarDist, custom-trained StarDist, and DeepLabV3. Because the two fixation methods were evaluated within the same patients, all comparisons were performed using paired observations.

The Shapiro–Wilk test provided no evidence against normality of the paired FFPE–CRYO differences for the pre-trained StarDist (p = 0.27), custom-trained StarDist (p = 0.28), or DeepLabV3 (p = 0.95) models. Accordingly, paired t-tests were used as the primary analysis, with Wilcoxon signed-rank tests performed as sensitivity analyses.

For the pre-trained StarDist model, the mean FFPE–CRYO difference was −0.003 (95% CI, −0.025–0.019; paired t-test, t(17) = −0.33, p = 0.75; Wilcoxon signed-rank test, p = 0.60; Cohen's d_z = −0.08). For custom-trained StarDist, CRYO yielded higher Dice coefficient than FFPE, with a mean paired difference of 0.0243 (95% CI, −0.0016–0.0503; paired t-test, t(17) = 1.98, p = 0.0641; Wilcoxon signed-rank test, p = 0.0483; Cohen's d_z = 0.47). For DeepLabV3, the mean paired difference was −0.015 (95% CI, −0.038–0.008; paired t-test, t(17) = −1.32, p = 0.21; Wilcoxon signed-rank test, p = 0.229; Cohen's d_z = −0.31).

Because the same comparison was performed for three model configurations, p-values were adjusted for multiple testing using Benjamini–Hochberg procedure. The adjusted p-values were 0.747, 0.192, and 0.345 for pre-trained StarDist, custom-trained StarDist, and DeepLabV3, respectively. None of the comparisons remained statistically significant after adjustment.

Overall, no consistent difference in segmentation performance between FFPE and CRYO was observed across the three model configurations. Although custom-trained StarDist showed a moderate effect favoring CRYO (Cohen's d_z = 0.68), this difference did not remain statistically significant after correction for multiple comparisons.

Discussion

In this study, we compared two DL architectures, StarDist and DeepLabV3, for automated nuclear segmentation of hematoxylin-stained lymphoid tissue prepared using two specimen-preparation methods: FFPE processing and CRYO. Three model configurations were evaluated: pre-trained StarDist, custom-trained StarDist, and DeepLabV3. Our results demonstrate complementary strength of the two approaches. DeepLabV3 achieved superior pixel-level overlap, whereas StarDist demonstrated higher PQ, more precise boundary localization, and greater stability under limited training data conditions. Both architectures showed broadly comparable performance across FFPE- and CRYO-prepared specimens. This observed differences were small and inconsistent in direction across architectures and evaluation metrics, suggesting that CRYO preparation is suitable for DL-based morphometric analysis rather than demonstrating superiority over FFPE. To our knowledge, few studies have systematically evaluated CRYO sections for automated nuclear segmentation of lymphoid tissue in this context. Both architectures have previously demonstrated strong performance across a range of biomedical imaging applications,1, 26, 27, 28, 29 whereas automated image analysis tools have been extensively applied to FFPE lymphoma specimens.30, 31, 32

The Dice coefficient was selected as the primary evaluation metric due to its widespread use in medical image segmentation. Reported values in histopathology typically range from approximately 0.71 to 0.95, with DL approaches in cancer applications commonly achieving approximately 0.84–0.89.33, 34, 35, 36, 37, 38, 39 In addition, complementary metrics, including F1 score, IoU, AJI, HD95, and PQ, were used to capture different aspects of segmentation performance, such as pixel-level overlap (F1, IoU), instance-level separation (AJI, PQ), and boundary precision (HD95). The values obtained in this study were broadly consistent with those reported for previous histopathological segmentation approaches.22, 23, 24, 25, 34, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49

A consistent discrepancy was observed between Dice coefficient (0.75–0.84) and PQ (0.55–0.61) across both architectures, reflecting the different aspects of segmentation captured by these metrics. Dice measures pixel-level overlap and may remain high when adjacent nuclei are merged, whereas PQ is sensitive to instance-level errors, including missed, merged, and split nuclei. This distinction is particularly relevant in densely packed lymphoid tissue, where limited inter-nuclear contrast may increase instance-level errors. Therefore, a high Dice coefficient alone may not be sufficient for applications involving nuclear counting or lymphocyte density estimation, and instance-aware metrics should also be considered. In our dataset, StarDist achieved higher PQ despite its lower Dice coefficient, suggesting more effective instance separation. However, this finding should be validated using task-specific measures of nuclear detection and counting accuracy in future studies.

The direct comparison of StarDist and DeepLabV3 demonstrated complementary segmentation characteristics. DeepLabV3 achieved substantially higher Dice and Jaccard values for both FFPE and CRYO specimens, indicating superior pixel-level foreground overlap. In contrast, StarDist yielded lower HD95 values and higher PQ values, suggesting more precise boundary localization and improved instance separation. The qualitative analysis supported these findings: in densely packed fields, DeepLabV3 generally recovered foreground pixels accurately but could merge adjacent nuclei during instance extraction, whereas StarDist more consistently delineated individual nuclei but occasionally missed faint or poorly focused objects. Thus, model selection may depend on the intended downstream application. DeepLabV3 may be advantageous when foreground-region segmentation is the primary objective, whereas StarDist may be preferable for instance-oriented applications such as nuclear counting.

The superior performance of DeepLabV3 with large training set can be partially attributed to its architectural design. Dilated (atrous) convolutions enable effective capture of multi-scale contextual information, improving representation of nuclei with heterogeneous size and morphology. Furthermore, the ASPP module enhances robustness to scale variation by aggregating features across multiple receptive fields. The use of an EfficientNet backbone further strengthens feature extraction and improves discrimination between cellular structures and background regions. In contrast, StarDist relies on star-convex shape priors, which facilitate efficient instance segmentation and contribute to its robustness under limited training data in our study, but may restrict its performance in highly irregular or densely overlapping cellular regions.18, 28, 29, 50, 51 Consistent with this interpretation, StarDist demonstrated stable performance across different train-test splits, whereas DeepLabV3 was more sensitive to training data size, with a noticeable decline in performance when the training set was reduced below 60%. This observation aligns with established machine learning principles, whereby higher-capacity models require larger datasets for optimal generalization. A similar stability across different train-validation-test split configurations has been reported in lymphoma image analysis using multiple CNN architectures.⁵² Dataset-specific factors may further influence performance, including the dense nuclear packing characteristic of lymphoid tissue and annotation variability.53, 54

The benefit of dataset-specific StarDist training differed between FFPE and CRYO specimens. Compared with the corrected pre-trained baseline, custom training produced only a small, non-significant improvement in Dice score for FFPE specimens, whereas a significantly larger improvement was observed for CRYO specimens. The limited improvement for FFPE may partly reflect the relatively high agreement of the pre-trained model with the corresponding annotation, as well as the suitability of StarDist's star-convex representation for the predominantly rounded lymphocyte nuclei examined in this study. In contrast, the greater improvement observed for CRYO suggests that dataset-specific training may provide greater adaptation to the morphological and imaging characteristics of these specimens. These findings should nevertheless be interpreted cautiously given the limited cohort size and should be validated in larger datasets using complementary instance-level metrics.

Segmentation of hematoxylin-stained images remains inherently challenging due to low contrast, high cellular density, tightly packed or even overlapping nuclei, and limited target specificity.⁵⁵ In contrast, immunofluorescence and immunohistochemistry provide high-contrast, target-specific signals that facilitate feature separation, and thus improve model performance.32, 56 Despite these challenges, the segmentation accuracy achieved in our study was broadly comparable with the reported recently histopathological segmentation approaches, including transformer-based models such as CellViT, dual-decoder U-Net models, and DCSANet.57, 58, 59 Direct numerical comparisons should, however, be interpreted cautiously because published studies differ in tissue type, staining protocol, image resolution, annotation strategy, and evaluation methodology.

Comparison of the results obtained by both DL architectures between FFPE and CRYO specimens revealed only minor differences in segmentation performance. Although slightly higher performance was observed for CRYO specimens in some configurations, no consistent advantage was observed across the evaluated models and metrics, and none of the specimen preparation method comparisons remained statistically significant after correction for multiple comparisons. Furthermore, taking into account that FFPE and CRYO sections were generated from different portions of each specimen and independently annotated, differences in tissue compositions and annotation variability may also have contributed to the performance differences observed. Overall, the comparable performance across both specimen preparation methods suggest that glyoxal-based CRYO preparation provides image quality suitable for DL-based segmentation.

Previous studies have shown that FFPE processing may introduce tissue shrinkage and structural artifacts,20, 60 whereas, glyoxal-based cryofixation may better preserve cellular morphology.16, 17, 60 These differences may be contributed to variation in tissue appearance; however, their specific effects on segmentation performance could not be fully isolated in the present study because, although FFPE and CRYO specimens were derived from the same lymph node, they were prepared from separate tissue fragments.

From a workflow perspective, CRYO preparation enables rapid slide generation (within 2–3 h) compared to the multi-day processing required for FFPE.20, 61, 62 CRYO preparation also avoids formalin fixation. Although glyoxal is itself an irritant and sensitizer and requires appropriate handling, glyoxal-based fixatives have been reported to present lower toxicity and reduced ventilation requirements compared with formalin-based fixatives.17, 63, 64 Thus, CRYO may offer certain occupational and workflow advantages over formalin-based processing when appropriate safety precautions are followed. In the present study, the comparable segmentation performance observed across FFPE- and CRYO-prepared specimens further supports the feasibility of CRYO for DL-based morphometric analysis.

Benchmarking against recently published nuclear-segmentation approaches provides additional context for the performance observed in this study. DeepLabV3 achieved Dice coefficients of 0.82–0.84 and AJI values of 0.62–0.63, indicating strong pixel- and instance-level segmentation performance.24, 48, 57, 58, 59, 65 Although StarDist produced slightly lower Dice (0.75–0.78) values, it achieved higher PQ (0.58–0.61) scores, reflecting improved instance separation and boundary precision.24, 58 Overall, these results are broadly comparable with those reported for contemporary nuclei-segmentation methods, including U-Net-based, dual-decoder, and transformer-based approaches.57, 58, 59

Several limitations of the current study should be considered. First, the cohort was relatively small and included limited representation of individual lymphoma subtypes, precluding reliable subtype-specific analysis. Second, FFPE and CRYO images were generated from different portions of each biopsy specimen and independently annotated; therefore, differences in tissue composition and annotation variability may also have contributed to the observed performance differences. Although the patient-level paired analysis accounts for between-patient variability, it does not eliminate section-level differences. Third, all images were acquired at a single institution using the same microscope and camera system. Consequently, the effects of inter-lab variability, alternative imaging platforms, staining protocols, and tissue-processing procedures should be evaluated in future studies to establish generalizability of the findings.

The complementary strengths of the two architectures should also be considered in the context of their respective limitations. StarDist relies on a star-convex representation and may therefore be less effective for highly irregular or substantially overlapping nuclei. DeepLabV3 achieved higher region-based accuracy but is more sensitive to reduction in training-set size in the present cohort. Future work should explore the integration of attention mechanisms or transformer-based architectures, which may improve segmentation of densely packed or morphologically heterogeneous nuclei.57, 58, 59 Domain adaptation strategies may also improve generalization across differences in staining, imaging platforms, and tissue-processing protocols.

The data-efficiency comparison should also be interpreted cautiously. The lowest training proportion corresponded to approximately four patients, and the comparison between StarDist and DeepLabV3 was not capacity-matched. In particular, the EfficientNet-B7 backbone used for DeepLabV3 has substantially greater model capacity than the StarDist architecture, which may contributed to the observed performance degradation under very limited training conditions. Therefore, the greater stability of StarDist should not be interpreted as definitive evidence of an intrinsic architectural advantage in data efficiency. A capacity-matched comparison using a lighter backbone, together with validation on larger independent cohorts, is required to determine whether this difference generalizes beyond the present dataset.

Future studies should evaluate segmentation performance using task-specific endpoints, particularly nuclear detection and counting accuracy, in addition to conventional pixel- and instance-level metrics. Larger, multi-institutional datasets encompassing a broader spectrum of lymphoma subtypes, staining conditions, and imaging platforms would enable more robust assessment of generalizability. Domain adaptation strategies may further improve performance across difference in staining, imaging systems, and tissue-processing protocols. Furthermore, attention mechanisms and transformer-based architectures, including recent approaches such as CellViT⁵⁷ and DCSANet,⁵⁹ may provide additional improvements for challenging cases involving densely packed or morphologically heterogeneous nuclei.

Overall, the results demonstrate complementary strengths of DeepLabV3 and StarDist. DeepLabV3 achieved superior pixel-level overlap, whereas StarDist demonstrated advantages in boundary localization, PQ, and robustness to reduction in training-set size. These findings suggest that model selection should be guided by the intended downstream application rather than by a single segmentation metric. Importantly, the comparable performance obtained with FFPE and CRYO preparations indicate that glyoxal-based CRYO processing can provide tissue morphology suitable for DL-based nuclear segmentation. CRYO therefore represents a promising alternative specimen preparation strategy for applications requiring rapid tissue processing and computational morphometric analysis, although validation in larger and independent cohorts is required before broader implementation.

Conclusions

We demonstrated that both StarDist and DeepLabV3 enable accurate nuclear segmentation of hematoxylin-stained lymphoid tissue prepared using FFPE and glyoxal-based CRYO protocols. DeepLabV3 achieved superior pixel-level and region-based segmentation accuracy, whereas StarDist showed greater robustness under limited training data and more accurate boundary and instance delineation. These complementary characteristics highlight a trade-off between region-level accuracy, instance-level segmentation, and data efficiency that should be considered when selecting a model for specific analytical applications.

Importantly, CRYO specimens demonstrated sufficient morphological preservation for reliable DL-based nuclear segmentation and represents a rapid alternative to conventional FFPE processing for quantitative histomorphometric analysis.

Overall, these findings support the feasibility of combining DL-based nuclear segmentation of CRYO specimens in computational pathology workflows. Validation in larger, multi-institutional cohorts, and across diverse tissue-processing and imaging conditions will be required to establish the generalizability of these findings.

Author contribution statement

A.T., A.B., A.I., and I..V.: writing—original draft preparation; A.K., I..V., and D.Y.: review and editing; A.K., I..V.: supervision; A.K., I..V.: funding acquisition. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan (Grant No. BR24993094) and by Nazarbayev University under the Faculty Development Competitive Research Grant Program (FDCRGP), Grant No. 040225FD4736, and the Faculty Development Competitive Research Grant Program (FDCRGP), Grant No. 201223FD8829.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.