Adaptive clustering for medical image analysis using the improved separation index

Abstract

Clustering high-dimensional biomedical data without prior knowledge of the number of clusters remains a major challenge in medical image and signal analysis. We present SONSC (Separation-Optimized Number of Smart Clusters), an adaptive and interpretable clustering framework driven by the Improved Separation Index (ISI)—a novel internal validity metric that jointly evaluates intra-cluster compactness and inter-cluster separability. SONSC iteratively maximizes ISI across candidate cluster configurations to automatically infer the optimal number of clusters, without supervision or parameter tuning. Extensive experiments on benchmark datasets (MNIST, CIFAR-10) and real-world clinical modalities (chest X-ray, ECG, RNA-seq) demonstrate that SONSC consistently outperforms classical methods such as K-Means, DBSCAN, and spectral clustering in ISI, Silhouette score, and normalized mutual information (NMI). Beyond numerical performance, SONSC identifies clinically coherent structures aligned with expert-labeled categories, supporting its integration into diagnostic and patient stratification pipelines. By unifying algorithmic robustness with medical interpretability, SONSC provides a scalable and trustworthy solution for unsupervised biomedical data analysis.

Introduction

The exponential growth of biomedical data—from multi-modal medical images to high-throughput omics profiles—has intensified the need for unsupervised learning methods capable of extracting meaningful patterns without access to labeled annotations. In clinical domains such as radiology, cardiology, and oncology, annotated data are often costly, error-prone, or impractical to obtain at scale. Consequently, clustering has become a foundational tool for applications such as disease subtyping, anomaly detection, risk stratification, and early screening^1,2.

Classical clustering algorithms—including K-Means, Hierarchical Clustering, and Gaussian Mixture Models (GMMs)—are valued for their simplicity and interpretability. However, they exhibit key limitations in clinical contexts: (1) the need to specify the number of clusters a priori, (2) sensitivity to noise, and (3) poor performance in high-dimensional, sparse, and non-convex feature spaces^3,4. These challenges are exacerbated by the heterogeneous nature of real-world biomedical data, where underlying patterns are often subtle and weakly expressed.

To mitigate these issues, several internal cluster validity indices (CVIs)—such as the Silhouette Score, Davies–Bouldin Index, and Dunn Index—have been proposed for unsupervised quality evaluation^5,6. While effective on idealized benchmarks, these metrics often fail in clinical settings due to irregular cluster shapes, label imbalance, and data noise^2,7.

Recent advances in deep clustering have sought to overcome these limitations by coupling representation learning with unsupervised objectives. Models such as Deep Embedded Clustering (DEC), and more recent frameworks like SCAN and DINO, leverage learned embeddings to improve clustering performance^8,9. However, these methods are typically opaque, computationally expensive, and sensitive to hyperparameters—rendering them less suitable for deployment in clinical workflows¹⁰.

In response, recent frameworks such as CDC¹¹ and MscNMF¹² have introduced new strategies to tackle the challenges of clustering in complex biomedical data. CDC (Complex Data Clustering) employs a simple yet effective optimization framework with generalizable assumptions, suitable for sparse and high-dimensional features. Meanwhile, MscNMF applies multi-structured non-negative matrix factorization for simultaneous clustering and gene marker extraction in scRNA-seq data, achieving biologically meaningful separability. Despite these advancements, many of these models require careful tuning or are limited in scalability and interpretability when applied across diverse clinical modalities.

Graph-based neural clustering methods (e.g., Graph Convolutional Networks) enhance representation quality by incorporating structural relationships between samples, showing promise in histopathological and whole-slide imaging¹³. Yet, their black-box nature and reliance on large labeled training corpora pose significant obstacles in safety-critical domains like healthcare.

Proposed solution

To address these limitations, we introduce SONSC (Separation-Optimized Number of Smart Clusters)—a novel, adaptive, and interpretable clustering framework tailored for biomedical data. SONSC is driven by the Improved Separation Index (ISI), a newly proposed internal validity metric that simultaneously measures intra-cluster compactness and inter-cluster separation in a noise-resilient and scalable manner. The method automatically infers the optimal number of clusters through an ISI-guided iterative process, requiring neither supervision nor manual tuning.

Research contributions

This study makes the following key contributions:

• We propose the Improved Separation Index (ISI), a robust and theoretically grounded metric for evaluating clustering quality in high-dimensional, sparse, and noisy biomedical datasets.

• We introduce SONSC, a parameter-free clustering algorithm that adaptively discovers the optimal number of clusters by maximizing ISI.

• We empirically validate SONSC on both synthetic benchmarks (e.g., MNIST, CIFAR-10) and real-world clinical datasets—including chest X-rays, ECG signals, and RNA-seq—demonstrating superior performance over classical and deep clustering baselines.

• We show that SONSC reveals clinically coherent cluster structures aligned with expert labels, supporting its integration into medical decision-support systems.

• We provide a reproducible and open-source implementation to facilitate adoption within biomedical AI pipelines.

Motivation and research gap

This work is motivated by the lack of clustering algorithms that are simultaneously adaptive, interpretable, and reliable in clinical environments characterized by noisy, unstructured, and high-dimensional data. Specifically, we aim to address the following gaps:

• Limited generalizability of existing methods across heterogeneous biomedical modalities and data distributions.

• The absence of stable and effective internal validation metrics suitable for real-world clinical complexity.

• The need for explainable clustering outcomes aligned with diagnostic categories and usable in high-stakes decision support.

SONSC bridges these gaps by combining deep feature embeddings with an ISI-guided adaptive clustering mechanism, delivering a scalable, interpretable, and clinically relevant solution for unsupervised biomedical data analysis. By unifying statistical rigor with clinical applicability, SONSC not only addresses existing limitations but also provides a generalizable framework for interpretable, unsupervised discovery in high-stakes biomedical environments.

Related work

Overview

CClustering remains a cornerstone of unsupervised learning, especially in biomedical domains where data is high-dimensional, noisy, and largely unlabeled. From disease subtyping to anomaly detection and personalized diagnostics, clustering enables downstream discovery across various modalities, including MRI, ECG, histopathology, and transcriptomics^14–16. A central challenge in these contexts is the automatic estimation of the optimal number of clusters and ensuring robustness to noise, sparsity, and irregular feature distributions. Existing methodologies can be broadly grouped into: (1) classical clustering algorithms, (2) internal cluster validity indices (CVIs), and (3) adaptive and learning-based clustering frameworks^5,17.

A central challenge in these contexts is the automatic estimation of the optimal number of clusters and ensuring robustness to noise, sparsity, and irregular feature distributions. Existing methodologies can be broadly grouped into: (1) classical clustering algorithms, (2) internal cluster validity indices (CVIs), and (3) adaptive and learning-based clustering frameworks^5,17.

Classical clustering algorithms

Traditional methods such as K-Means, Hierarchical Clustering, and DBSCAN remain widely used due to their simplicity and computational speed. K-Means is efficient but sensitive to initialization and assumes spherical clusters, making it suboptimal for biomedical data¹⁸. Hierarchical clustering offers visual interpretability but does not scale well and is susceptible to noise¹⁹. Density-based methods like DBSCAN and OPTICS improve flexibility but often require domain-specific hyperparameter tuning²⁰.

Hybrid approaches attempt to address these issues. For example, MedTransCluster combines pretrained CNN features with K-Means to enhance medical image segmentation performance²¹. Nonetheless, these models typically lack adaptability and transparency in real-world settings.

Cluster validity indices (CVIs)

CVIs play a crucial role in unsupervised model selection and quality evaluation. Common indices include:

• Silhouette Score: Evaluates separation and compactness but fails in high-dimensional or overlapping clusters²².

• Davies–Bouldin Index: Sensitive to noise and assumes convex cluster geometry²⁰.

• Dunn Index: Emphasizes inter-cluster separation but is computationally expensive²³.

• Cosine Clustering Index (CCI): Adapted for cosine similarity spaces, yet remains underexplored in clinical settings²⁴.

Hassan et al.⁶ conducted a comprehensive review of over 60 CVIs and emphasized the lack of robust, scalable, and interpretable metrics suitable for biomedical data. This motivates the development of newer indices such as our proposed Improved Separation Index (ISI), tailored for clinical use.

Modern adaptive clustering techniques

Contemporary clustering frameworks leverage deep architectures, graph models, and self-supervised learning to improve flexibility and performance:

• CDC Framework: Kang et al.¹¹ propose CDC, a simple yet effective clustering pipeline that generalizes well to heterogeneous and high-dimensional datasets with minimal tuning.

• MscNMF: Wang et al.¹² present a multi-structured non-negative matrix factorization framework that jointly performs clustering and gene marker extraction, enhancing interpretability in genomics.

Despite these advances, most adaptive techniques fall short in producing human-interpretable clusters, handling noise gracefully, or providing internal validation without labels—three core requirements in clinical AI pipelines.

Sparse and Factorization-based Clustering Models. Several recent approaches have specifically addressed intra-cluster sparsity and inter-cluster separation using matrix factorization techniques. For instance, MscNMF¹² imposes structured sparsity constraints on both the basis and coefficient matrices during non-negative matrix factorization, enabling more interpretable and compact cluster representations—especially in high-dimensional biomedical data. These models are particularly effective when clusters are expected to exhibit low-rank or latent sparse characteristics, such as in gene expression and single-cell omics data. While SONSC does not rely on explicit factorization, its ISI-driven formulation captures similar structural assumptions implicitly, without requiring initialization or manual regularization.

Positioning of SONSC

To bridge these gaps, we introduce SONSC: a noise-resilient, interpretable, and parameter-free clustering framework powered by the Improved Separation Index (ISI). Unlike prior approaches, SONSC:

• Automatically estimates the number of clusters via ISI maximization;

• Maintains strong performance in noisy and high-dimensional environments;

• Generates clinically explainable cluster assignments without supervision.

Table 1 summarizes the comparative performance of SONSC against baseline clustering algorithms.

Table 1.

Comparison of SONSC with classical and adaptive clustering methods.

Method	Requires	Noise robustness	Interpretability	Computational load
K-Means	Yes	Low	Moderate	Low
DBSCAN	No	High	Low	High
GMM	Yes	Moderate	Low	Moderate
Autoencoder clustering	No	Moderate	Low	Very high
CDC11	No	High	Moderate	Moderate
MscNMF12	No	Moderate	High	High
SwAV25	No	High	Low	Very high
SONSC (Proposed)	No	High	High	Moderate

As illustrated in Fig. 1, SONSC is positioned favorably in terms of adaptability, interpretability, and scalability compared to classical and deep clustering methods.

Fig. 1.

Positioning of SONSC in the clustering landscape: balancing adaptability, interpretability, and scalability.

Summary

With increasing reliance on machine learning for clinical decision support, clustering frameworks must evolve to offer resilience, explainability, and automation. SONSC satisfies these criteria by unifying a noise-aware internal validation metric (ISI) with adaptive clustering, enabling scalable and clinically meaningful insights from complex biomedical datasets.

Proposed SONSC algorithm based on the improved separation index (ISI)

Clustering is a cornerstone of unsupervised learning, enabling the extraction of latent structures from data in the absence of explicit labels. This capability is particularly critical in biomedical domains, where annotated datasets are often scarce, high-dimensional, noisy, and heterogeneous. While traditional clustering methods—such as K-Means—are widely used due to their simplicity, they suffer from major limitations, including the requirement to predefine the number of clusters and suboptimal performance on non-convex or complex distributions²⁶.

To address these challenges, we propose SONSC (Selection of Optimal Number of Smart Clusters), a novel, interpretable, and domain-agnostic clustering framework. At its core lies the Improved Separation Index (ISI)—a robust internal cluster validity metric that jointly captures intra-cluster compactness and inter-cluster separability. By evaluating ISI across a range of candidate cluster numbers , SONSC dynamically determines the optimal number in a data-driven and parameter-free manner.The overall pipeline of the proposed SONSC framework is depicted in Fig. 2.

Fig. 2.

Schematic overview of the SONSC framework. Feature vectors are extracted from biomedical data using pretrained neural encoders. K-Means++ is iteratively applied across increasing values of , and clustering quality is evaluated via the Improved Separation Index (ISI). The process halts when ISI ceases to improve, yielding the optimal number of clusters .

Core contributions

SONSC introduces the following key innovations:

• Improved Separation Index (ISI): A mathematically grounded validity index that balances cohesion and separation, designed to maintain stability in noisy, sparse, and high-dimensional datasets.

• Automatic Cluster Discovery: SONSC leverages a progressive ISI-based search to identify the optimal cluster count without relying on heuristics or manual tuning.

• Cross-Modality Generalization: The framework is modality-agnostic, with demonstrated applicability across imaging (MRI, X-ray), signal (ECG), and molecular (RNA-seq) datasets.

Methodological framework

SONSC proceeds through three primary phases:

Feature representation

Samples are encoded into compact feature vectors using pretrained convolutional networks (e.g., ResNet50), followed by normalization. Optionally, dimensionality reduction via Principal Component Analysis (PCA) is applied to denoise and enhance computational efficiency.

ISI-based cluster evaluation

For each candidate cluster count , K-Means++ is applied, and clustering quality is assessed via the Improved Separation Index:

1

where cohesion and separation are defined as:

2

3

Unlike traditional metrics such as the Silhouette or Davies-Bouldin indices, ISI offers a unified and interpretable formulation that is resilient to noise, class imbalance, and high dimensionality.

Adaptive cluster discovery

Starting from , SONSC incrementally increases the number of clusters. At each step, ISI is recalculated. The process continues until ISI no longer improves, at which point the current best is selected.

SONSC algorithm

Algorithm 1.

SONSC: Adaptive Clustering via Improved Separation Index

Theoretical advantages

SONSC offers several theoretical and practical benefits:

• High-Dimensional Robustness: ISI remains stable and discriminative in complex, sparse feature spaces.

• Noise Resilience: Global centroid-based separation enhances resistance to local noise and outliers.

• Interpretability: Each component—clustering, validation, and selection—is transparent and modular, supporting clinical use cases.

• No Hyperparameter Tuning: The adaptive process is fully automated and requires no manual input or prior knowledge.

Limitations and future directions

Despite its effectiveness, SONSC has certain limitations:

• Metric Dependency: The current reliance on Euclidean distance may limit effectiveness on non-linear manifolds. Future work could incorporate kernelized or geodesic extensions.

• Computational Overhead: Evaluating ISI across many cluster counts can be expensive for large-scale datasets. Efficient approximations and early stopping heuristics are potential improvements.

• Outlier Sensitivity: As a centroid-based method, SONSC may be influenced by extreme values. Robustified statistics could mitigate this.

Summary

In summary, SONSC introduces a novel, interpretable, and mathematically grounded framework for unsupervised clustering in biomedical contexts. By leveraging the Improved Separation Index (ISI), SONSC provides an automated and scalable solution that balances clustering performance with clinical interpretability—offering a practical path for integration into diagnostic, screening, and precision medicine pipelines.

Experimental results on real-world medical datasets

To evaluate the effectiveness, robustness, and generalizability of the proposed SONSC framework, we conducted extensive experiments on three representative biomedical datasets. These datasets span multiple modalities—imaging (MRI), electrophysiology (ECG), and transcriptomics (RNA-seq)—capturing the diversity of real-world medical applications. Table 2 summarizes the datasets and their corresponding clinical tasks.

Table 2.

Biomedical datasets used for SONSC evaluation and their clinical relevance.

Dataset	Modality	Clinical objective
BraTS 2020	Brain MRI (T1, T2, FLAIR)	Tumor subregion segmentation
MIT-BIH arrhythmia	ECG time-series	Arrhythmia classification
TCGA-BRCA	RNA-seq gene expression	Breast cancer molecular subtyping

TCGA-BRCA: unsupervised molecular subtyping in breast cancer

extcolorredThe TCGA-BRCA dataset provides high-dimensional RNA-seq profiles of breast cancer patients. As a clinically annotated, genomics-scale dataset, it offers a stringent benchmark for unsupervised molecular subtyping.

Optimal cluster discovery via ISI

We applied SONSC to the RNA-seq expression profiles from the TCGA-BRCA dataset. The Improved Separation Index (ISI) was computed for . As shown in Fig. 3, ISI peaked at , suggesting that the gene expression landscape contains eight biologically meaningful subtypes.

Fig. 3.

ISI, cohesion, and separation scores for TCGA-BRCA. Optimal cluster count identified at .

Biological relevance and gene-level differentiation

We conducted one-way ANOVA on the rs_TFF1 biomarker, which plays a pivotal role in estrogen receptor-positive breast cancer. The expression levels differed significantly across clusters (, ), as illustrated in Fig. 4.

Fig. 4.

Expression of rs_TFF1 across SONSC clusters in TCGA-BRCA. Distinct expression patterns support biological separability.

Alignment with histological labels

The SONSC clusters were compared with expert-annotated histological subtypes (e.g., ductal and lobular carcinoma). As shown in Fig. 5, SONSC effectively distinguishes between major tumor classes, achieving a Normalized Mutual Information (NMI) score of 0.76.

Fig. 5.

Heatmap: SONSC clusters vs. histological subtypes in TCGA-BRCA. Clear alignment with clinical categories.

MIT-BIH: unsupervised arrhythmia pattern discovery from ECG signals

ISI-based optimal cluster selection

We extracted temporal and morphological features (e.g., RR interval, QRS width) from 48 ECG records and applied SONSC. The ISI score increased until and dropped thereafter (Fig. 6), indicating as the optimal cluster count.

Fig. 6.

ISI vs. number of clusters for MIT-BIH ECG dataset. SONSC determines as optimal.

Feature-based cluster interpretability

A boxplot analysis of RR intervals across clusters (Fig. 7) revealed significant physiological differences (, ), supporting SONSC’s discriminative capability in the context of cardiac rhythm variations.

Fig. 7.

RR interval distribution across SONSC clusters. Variability supports detection of different arrhythmias.

Comparison with expert annotations

As shown in Fig. 8, SONSC clusters correspond closely with known arrhythmic classes (e.g., PVC, LBBB), revealing clear physiological alignment despite being learned in an unsupervised manner.

Fig. 8.

Heatmap: SONSC clusters vs. annotated ECG beat types. Clear differentiation of pathological and normal rhythms.

BraTS 2020: tumor subregion segmentation in brain MRI

Optimal region discovery via ISI

We applied SONSC on FLAIR-based feature vectors extracted from the BraTS MRI dataset. The ISI score peaked at (Fig. 9), identifying distinct tumor subregions including enhancing core, edema, and necrotic zones.

Fig. 9.

ISI score versus cluster number for BraTS MRI. Optimal tumor segmentation found at .

Tissue-level signal variation

Figure 10 shows the distribution of FLAIR intensity across clusters. A one-way ANOVA confirmed significant inter-cluster variability (, ), which supports anatomical distinctiveness of tumor compartments.

Fig. 10.

Boxplot of mean FLAIR intensity across MRI clusters. Clusters reflect distinct tumor tissue properties.

Alignment with ground-truth segmentation

Comparison with expert annotations shows that the SONSC-derived clusters align closely with tumor subregions (Fig. 11), verifying clinical utility.

Fig. 11.

Heatmap: SONSC clusters vs. expert tumor labels. Accurate mapping of edema, necrosis, and enhancing tumor.

Cross-dataset performance summary

Statistical comparison of features

Table 3 summarizes the ANOVA results across all datasets. All features showed statistically significant differences across SONSC clusters (), affirming the model’s discriminative power.

Table 3.

ANOVA results for key biomarkers across SONSC clusters.

Dataset	Feature	F-statistic	p-value
MIT-BIH (ECG)	RR interval (sec)	72.43
TCGA-BRCA	rs_TFF1 expression	155.07
BraTS MRI	Mean FLAIR intensity	48.92

Quantitative clustering metrics

As presented in Table 4, SONSC outperforms classical clustering techniques across ISI and silhouette metrics, and achieves highest NMI for the labeled BRCA dataset.

Table 4.

Quantitative clustering results across medical datasets.

Dataset	Method	ISI	Silhouette	NMI
BraTS MRI	K-Means	0.72	0.48	–
	Spectral Clustering	0.75	0.53	–
	SONSC	0.89	0.71	–
MIT-BIH ECG	DBSCAN	0.63	0.51	–
	Agglomerative	0.61	0.47	–
	SONSC	0.87	0.68	–
TCGA-BRCA	K-Means	0.68	0.54	0.60
	Spectral Clustering	0.72	0.58	0.63
	SONSC	0.88	0.69	0.76

Cluster dispersion visualization using t-SNE

To further assess the separability and latent structure of the SONSC-discovered clusters, we applied t-Distributed Stochastic Neighbor Embedding (t-SNE) to project high-dimensional feature vectors into two-dimensional space. This visualization highlights how well the clusters learned by SONSC occupy distinct regions in the embedding space, providing intuitive insight into inter-cluster boundaries and intra-cluster cohesion.

TCGA-BRCA (k=8)

As shown in the leftmost panel of Fig. 12, the SONSC-derived clusters form well-separated and compact groupings in the gene expression space. Despite the intrinsic complexity and noise typical of transcriptomic data, SONSC uncovers eight clusters with minimal overlap—underscoring the algorithm’s effectiveness in molecular subtype discovery.

Figure 12.

t-SNE-based visualization of cluster dispersion across the three biomedical datasets. Each point represents a sample projected into two-dimensional space, color-coded by SONSC-assigned cluster.

MIT-BIH ECG (k=9)

The middle panel presents the t-SNE projection for the ECG dataset. Although some clusters appear closer due to natural overlaps in rhythm features (e.g., between normal and certain arrhythmic patterns), SONSC still achieves clear separation in several clusters, reflecting distinct cardiac signal morphologies. This is especially meaningful given the large sample size (10,000 beats) and time-series nature of the data.

BraTS MRI (k=8)

In the rightmost panel, clusters derived from deep feature embeddings of brain tumor MRI scans demonstrate strong spatial isolation. Each cluster corresponds to distinct tumor compartments (e.g., necrotic core, enhancing rim), consistent with anatomical regions known from clinical segmentation maps.

Interpretation. The t-SNE visualizations offer strong empirical support for the discriminative power of SONSC in high-dimensional biomedical domains. The clusters exhibit low intra-cluster variance and high inter-cluster separation, validating both the quality of the feature space and the adaptive clustering strategy.

Ablation study on ISI utility

Removing ISI-based stopping from SONSC led to over-segmentation and reduced clustering quality (Fig. 13). This confirms the importance of ISI as an adaptive and interpretable control mechanism.

Fig. 13.

Ablation: SONSC without ISI shows degraded performance.

Conclusion. Overall, SONSC consistently discovers meaningful, biologically interpretable clusters across diverse clinical datasets. Its parameter-free, adaptive design makes it suitable for a wide range of unsupervised medical AI tasks.

Evaluation on the leukemia gene expression dataset

To evaluate the performance of SONSC on a standard biomedical benchmark, we applied the algorithm to the Leukemia dataset introduced by Golub et al.²⁷, consisting of 72 samples (AML and ALL subtypes) with expression levels for 7129 genes. The dataset poses significant challenges for unsupervised methods due to its small sample size and high-dimensional structure.

We initialized SONSC with and incremented the cluster count until the Improved Separation Index (ISI) reached its peak. Table 5 summarizes the ISI scores for each value of .

Table 5.

ISI scores computed by SONSC across candidate values of on the Leukemia dataset.

Number of Clusters ()	ISI score
2	0.0154
3	0.0271
4	0.0379
5	0.0513
6	0.0679
7	0.0826
8	0.1035
9	0.1210
10	0.1371

As depicted in Fig. 14, the optimal number of clusters was found to be , corresponding to the peak ISI score of 0.1371. The clustering results were visualized using a 2D projection via Principal Component Analysis (PCA), revealing compact subgroups despite the biological sparsity.

Fig. 14.

PCA projection of SONSC-derived clusters on the Leukemia dataset. SONSC automatically estimated clusters using ISI.

Although the traditional interpretation of this dataset suggests 2–3 main classes, SONSC revealed finer-grained substructure. This aligns with its design for unsupervised exploration, particularly in noisy biomedical contexts. A follow-up SHAP-based feature attribution in section "Feature attribution via SHAP analysis" validates that the clusters are not only structurally distinct but biologically meaningful as well.

Baseline comparison with PCA + KMeans (DINO-inspired)

To benchmark SONSC against a standard deep clustering baseline, we constructed a DINO-like pipeline using PCA for feature extraction followed by KMeans clustering. Table 6 reports the Normalized Mutual Information (NMI) and ISI scores across the same range of . Figure 15 visualizes this comparison.

Table 6.

Comparison of DINO-like baseline (PCA+KMeans) with SONSC using ISI and NMI scores.

	NMI (baseline)	ISI (baseline)
2	0.2516	0.3559
3	0.3794	0.4514
4	0.4568	0.5378
5	0.5160	0.5998
6	0.5602	0.6393
7	0.6060	0.6733
8	0.6296	0.7286
9	0.6518	0.8121
10	0.6834	0.8294

Fig. 15.

ISI and NMI progression for the DINO-like baseline (PCA+KMeans) across varying values of . While ISI increases steadily, the interpretability and robustness of clusters are not guaranteed.

While the baseline model achieves high NMI and ISI scores at , SONSC retains its advantage in automatic cluster number detection, model transparency, and clinical interpretability (see sections "Feature attribution via SHAP analysis" and "Discussion"). Hence, this comparison validates the relevance of ISI-guided optimization and the broader utility of SONSC in biomedical clustering tasks.

Discussion

The comprehensive evaluation of SONSC across synthetic datasets (e.g., spirals, concentric circles), widely used benchmarks (MNIST, CIFAR-10), and real-world clinical datasets (BraTS MRI, ECG signals, TCGA-BRCA) highlights its robust and adaptive clustering capabilities. In all tested scenarios, SONSC consistently outperformed conventional and modern clustering algorithms in terms of ISI, silhouette score, and, where applicable, NMI.

A notable strength of SONSC lies in its capacity to uncover latent structures without the need for prior labels or manually specified cluster counts. This is particularly advantageous in medical applications, where data is often high-dimensional, noisy, and imbalanced. For example, in the BraTS MRI dataset, SONSC successfully differentiated between subtle tumor subregions (e.g., edema, necrosis), and in TCGA-BRCA, it aligned strongly with known molecular breast cancer subtypes. These findings emphasize the algorithm’s practical utility and translational relevance.

The key to this robustness is the Improved Separation Index (ISI), which effectively balances intra-cluster cohesion with inter-cluster separation. ISI also offers a stable and reliable stopping criterion that supports adaptive determination of the optimal number of clusters. Compared to existing internal validation indices, ISI demonstrated better sensitivity and stability, particularly in heterogeneous and complex datasets. Importantly, SONSC preserves transparency and computational efficiency, positioning it as a pragmatic alternative to deep clustering frameworks—especially in clinical scenarios requiring explainability and reproducibility.

Robustness to outliers

Although SONSC is a centroid-based clustering approach, which may conventionally be sensitive to outliers, its integration with the Improved Separation Index (ISI) helps mitigate this issue. Specifically, ISI penalizes cluster configurations where isolated or extreme points disproportionately affect intra-cluster distances. In our experiments (see Supplementary Figure S2), we introduced synthetic noise and extreme values to several datasets including Leukemia, and observed minimal changes in the final clustering assignments and ISI scores. This empirical robustness suggests that SONSC is relatively stable even in the presence of noisy biomedical data, such as electronic health records or high-variance omics features. However, further exploration using robust distance metrics or preprocessing techniques like outlier filtering could enhance resilience in extremely noisy domains.

Computational complexity and runtime evaluation

To assess the computational cost of SONSC, we benchmarked its average runtime on three datasets against standard KMeans and a DINO-inspired baseline (i.e., PCA followed by KMeans). All experiments were conducted on a standard laptop (Intel Core i7-1165G7 CPU, 16GB RAM, no GPU). The reported times represent average wall-clock durations over five runs.

As shown in Table 7, SONSC requires modest additional time due to its iterative ISI-guided mechanism. However, the total runtime remains within practical bounds even on large-scale datasets, with no need for GPU resources or deep learning frameworks.

Table 7.

Runtime comparison (in seconds) across different clustering algorithms.

Dataset	KMeans	DINO-like (PCA+KMeans)	SONSC
MNIST (10,000–784)	3.5	4.9	7.6
TCGA-BRCA (1100–5000)	1.8	3.2	4.9
Leukemia (72–7129)	0.08	0.12	0.21

In contrast to deep clustering methods requiring backpropagation and fine-tuning, SONSC offers a favorable tradeoff between interpretability, scalability, and runtime efficiency—making it well-suited for real-time clinical settings.

Conclusion

This work introduced SONSC, a novel and interpretable clustering framework that integrates deep representation learning with an internal self-regulation mechanism based on the Improved Separation Index (ISI). Unlike traditional clustering approaches, SONSC adaptively determines the number of clusters and produces meaningful groupings in an unsupervised manner.

Our experiments across diverse data types—ranging from medical imaging and physiological signals to genomics—demonstrated the framework’s superior performance in terms of clustering quality, adaptability, and domain alignment. SONSC not only discovered structure in unlabeled data but also achieved strong agreement with biological and clinical categorizations, confirming its applicability in real-world settings.

Overall, SONSC offers a compelling bridge between algorithmic rigor and clinical practicality. It provides a scalable, modality-agnostic, and interpretable solution for tasks such as patient stratification, tumor subregion segmentation, and biomarker discovery in the absence of labeled data.

Feature attribution via SHAP analysis

To further interpret the clustering outcomes of SONSC, we employed SHapley Additive exPlanations (SHAP), a model-agnostic method for feature attribution. A random forest classifier was trained using the cluster assignments generated by SONSC on the Leukemia gene expression dataset. SHAP values were then computed to quantify the contribution of each gene to the predicted cluster labels.

Figure 16 presents the SHAP summary plot, highlighting the top contributing genes across samples. Each bar represents a gene’s mean absolute SHAP value, indicating its relative importance in distinguishing clusters. The color gradient reflects the cluster affiliation of corresponding samples. Notably, distinct samples (e.g., Sample_4, Sample_10) exhibit strong gene-level associations with specific clusters, demonstrating the biological relevance of the clustering outcome.

Fig. 16.

SHAP summary plot illustrating gene-level feature contributions for SONSC cluster assignments on the Leukemia dataset. Each bar represents the average importance of a gene across all samples.

This analysis confirms that SONSC not only discovers coherent cluster structures but also enables the identification of discriminative biomarkers, thereby enhancing interpretability and clinical applicability in high-dimensional gene expression data.

Limitations

Despite its strengths, SONSC has several limitations that warrant further investigation:

• Dependence on Euclidean Distance: ISI relies on Euclidean distance for calculating cohesion and separation. This may limit performance on datasets with non-linear manifolds or non-metric structures. Future work could explore kernelized or geodesic-based distance functions.

• Scalability Concerns: The iterative search over increasing values of introduces additional computational overhead, particularly for large-scale datasets. Optimizations such as early stopping, subsampling, or distributed processing could enhance scalability.

• Outlier Sensitivity: Like other centroid-based methods, SONSC is affected by outliers that may skew cluster centers. Incorporating robust statistics or preprocessing steps (e.g., outlier removal or noise-aware training) may improve stability.

• Lack of Prior Knowledge Incorporation: While full unsupervised learning is a strength, certain clinical applications may benefit from weak supervision or domain-specific priors. Extending SONSC to semi-supervised scenarios is a promising direction.

Future work

Building on the promising foundation established by SONSC, several directions can further extend its capabilities:

• Kernelized and Graph-Based Variants: Adapting ISI to work with kernel or graph-based representations (e.g., via Laplacian embeddings or diffusion distances) may improve performance on manifold-structured biomedical data.

• Integration with Self-Supervised Learning: Embedding SONSC into self-supervised paradigms—such as contrastive or masked modeling—could enhance its representational depth while preserving unsupervised operation.

• Real-Time and Federated Extensions: Expanding SONSC for use in streaming, real-time, or federated learning scenarios (e.g., across distributed hospital networks) would improve clinical scalability and privacy compliance.

• Explainability Enhancements: Introducing post-hoc interpretation modules (e.g., SHAP-based feature attribution, concept activation maps, or cluster profiling) would further strengthen SONSC’s adoption in explainable AI frameworks.

These enhancements could transform SONSC into a fully modular, interpretable, and domain-adaptable unsupervised learning tool—ideally suited for diagnostic and prognostic decision-support in healthcare and beyond.

Data availability

All datasets used in this study are publicly available: - TCGA-BRCA RNA-seq data: https://portal.gdc.cancer.gov/projects/TCGA-BRCA - MIT-BIH ECG dataset: https://physionet.org/content/mitdb/1.0.0/ - BraTS 2020 MRI dataset: https://www.kaggle.com/datasets/awsaf49/brats2020-training-data.

Code availability

The implementation of the proposed SONSC algorithm is available at: https://github.com/Mojtaba-jahanian/SONSC-Biomedical-AutoClustering.

Declarations

Competing interests

The authors declares no competing interests.

Ethics approval

This research does not involve any human participants, animals, or sensitive data requiring ethical approval. All datasets used in this study are publicly available and were used in accordance with their respective data usage licenses.