A new possibilistic-based clustering method for probability density functions and its application to detecting abnormal elements

Abstract

Cluster analysis can also detect abnormalities besides building a basis for identifying elements into clusters. Detecting abnormalities is a highly developed feature in the field of unsupervised learning. However, existing studies have mainly focused on discrete data, not probability density functions. This paper enables a possibilistic approach to solving the clustering for probability density functions dealing with abnormal elements. First, the data are extracted using the density function. Then, they are passed through the proposed algorithm to produce a possibilistic partition. Finally, a decision rule is established to recognize which function is abnormal. We compare the proposed algorithm with baseline algorithms in clustering PDFs, such as k-means, FCF, and Self-Updated Clustering. The results of three numerical examples applied to the image are typical for this new method. Furthermore, The proposed algorithm reaches accuracy at 100% over simulated benchmark data and outperforms baseline methods. Additionally, two last examples apply to image data reaching G-mean up from 96 to 100% (Sensitivity: 92–100% and Specificity: 100%). The proposed method can be researched and used to understand the internal structures of big data in the digital age through the probability density functions.

Introduction

Data storage, extraction, and recognition pose crucial challenges for scientists in the information age¹. For recognition problems, clustering is a fundamental aspect of data mining and a well-established statistical technique for storing, categorizing, and organizing data^2,3. Clustering is one of the techniques to divide objects into disjoint and nonempty subgroups that transmit similar intrinsic characteristics⁴. Due to the development in terms of efficiency and accuracy of clustering algorithms, clustering has been applied extensively in various fields, such as in bioinformatics, microbiology, image segmentation/compression, geographical analysis, and any field requiring grouping similar features^5,6. Consequently, proposing new clustering algorithms and improving the performance of the existing ones have attracted the attention of statisticians and data scientists in diverse areas^7,8.

Over the past few years, tremendous research efforts have been made to evolve the partitioned clustering that seeks to obtain a single partition in a fixed number of clusters by optimizing an objective function. There are three main empirical philosophies, namely the hard (crisp, rigid, deterministic) clustering philosophy⁹ k-means, the soft (fuzzy, probabilistic) clustering philosophy^10–13 fuzzy c-means (FCM), furthermore, the possibilistic philosophy^14,15 (PCM). In hard clustering, each element belongs to a single cluster, so that does not give a probability of assigning; it might make some mistakes for boundary elements. Then, Dunn proposes the FCM¹⁶ based on Zadeh’s fuzzy theory¹⁷ and generalized by Bezdek^10,12 can be assigned to multiple clusters with different membership degrees across classes which sum to 1. However, k-means and FCM are highly sensitive to noise, which appears inconsistent with the remainder of that data set¹⁸. For the possibilistic approach, PCM relaxed the probabilistic constraint imposed on the membership value and overcame the effects of noise elements. Naturally, the PCM of Krishnapuram becomes a prominent technical advantage and is pioneering in clustering abnormal data.

An in-depth clustering analysis can be executed for primary types of data storage: clustering for probability density functions (CDF) besides discrete elements (CDE) and interval-valued (CDI)^19–22. When dealing with data that is volatile, uncertain, and vague, CDF data storage is more effective than CDE data storage²³. This is due to the fact that CDF data offers a comprehensive distribution of complex data at a low computational cost²⁴. Probability Density Function (PDF) data (or, more broadly, functional data²⁵ or signals²⁶) also has many applications in available analysis dealing with big data. The CDF problem comprises a range of almost hitherto mentioned algorithms, an array of ideas on traditional CDF algorithms consisting of non-fuzzy^27,28 and fuzzy approaches. The pioneering study on non-fuzzy CDF²⁹ reduced the clustering of multiple sub-manifolds on the unit Hilbert sphere to cluster a low-dimensional representation of PDFs. Hierarchical clustering of³⁰ employed the mixture models for hyper-spherical data.^31–33 established the cluster width criterion via the integral of the maximum function of PDFs and applied it for crisp clustering. Otherwise, for fuzzy clustering, Nguyentrang and Vovan contributed two fuzzy algorithms in CDF^1,34,35 with Seft Update^36–38 combined have introduced a method to determine the number of clusters and establish automatic optimal clusters. Even more extreme developments, the state-of-the-art CDF methods of the past half-decade do not adequately account for interference even though they became complicated, such as balanced drivers³⁹, semi-supervised⁴⁰, globally automatic⁴¹, efficient automatic fuzzy clustering⁴², and genetic algorithms^43,44 are both new methods and deal well with PDFs but do not address the problem of image abnormalities. Fuzzy algorithms for CDF are radicalized, even though they are stuck in noise sensitivity problems^19,45–47. Outliers can significantly affect the clustering process and the resulting partition, highlighting the need for a more incorporated outlier detection accurate final partition. To the best of our knowledge, no papers to date have proposed a possibilistic approach clustering method for CDF or a solution to the CDF problem involving the presence of abnormal elements. To address the above drawback, this study proposes a new possibilistic clustering algorithm for probability density functions (PCF) to account for the ability to identify outliers by combining ${\mathcal{L}}_1$-distance⁴⁸ with the idea of Krishnapuram¹⁴. The contributions are detailed as follows.

1. This study proposes a new robust and accurate clustering called The Possibilistic Clustering for probability density Functions (or PCF) extract by different estimations has been tested and compared with state-of-the-art algorithms.

2. This study provides a solid theoretical basis for developing PCF by Theorems 1 and 2.

3. This study proposes a method to detect both noises and abnormalities in PDFs by the threshold $\alpha$-cut.

4. The efficiency of the new framework is demonstrated through several examples of benchmarks and image data.

In this paper, we have achieved remarkable results in clustering images by applying our proposed algorithm for three numerical examples. Its stability and accuracy are presented, opening up new directions in cluster analysis to detect abnormal PDFs.

The remainder sections of this paper will be methodically arranged in the following. “Preliminary” section reviews some preliminary theories related to the clustering for PDF. The new possibilistic-based clustering algorithm is presented in “The proposed algorithm” section. Some numerical examples are investigated in “Numerical examples” section. “Conclusion remarks” section is the conclusion.

Preliminary

${\mathcal{L}}_1$-Distance for evaluating difference between PDFs

A similarity measure is required to detect the homogeneity/heterogeneity between two PDFs. The ${\mathcal{L}}_1$-distance is a well-established concept robust and fulfils the assumption criteria for a distance measure^48–50. The ${\mathcal{L}}_1$ is defined as follows.

Definition 1

(${\mathcal{L}}_1$-distance) A vector space $\mathcal{C}(\mathrm{\Omega })$, the all continuous complex functions on some open set $\mathrm{\Omega }$ in a Euclidean space ${\mathbb{R}}^d$. Let $f_1(\mathbf{x})$ and $f_2(\mathbf{x})$ be two PDFs defined on $\mathcal{C}(\mathrm{\Omega })$. The ${\mathcal{L}}_1$-distance between two functions is defined by Thu Pham-Gia⁴⁸

$$\begin{array}{c}\hfill \Vert f_1-f_2{\Vert }_1={\int }_{{\mathbb{R}}^d}|f_1(\mathbf{x})-f_2(\mathbf{x})|\text{d}\mathbf{x}.\end{array}$$ 1

${\mathcal{L}}_1$-distance has been used in works such as^1,36,39 to address clustering. This study also employs the ${\mathcal{L}}_1$-distance criterion to identify clusters.

Some validity measure indexes

Abnormals can cause problems for binary classification models that learn from noisy patterns and give biased results. Hence, this study’s bi-class evaluation metrics for anomalous data are described in Table 1. These metrics provide a comprehensive view of clustering model performance, supporting improved understanding and confidence in evaluating and comparing further clustering methods.

Table 1.

Some validity measure indexes used in this study.

Measure	Formula	Range	Status	Evaluation
Sen.	$TP/(TP+FN)$	[0, 1]	Larger is better	Accurately predict which elements are abnormals
Spec.	$TN/(TN+FP)$	[0, 1]	Larger is better	Accurately predict which elements are not abnormals
G-mean51	$\sqrt{\text{Sen.}\times \text{Spec.}}$	[0, 1]	Larger is better	A geometric mean of the sensitivity and specificity
F1-score	$2TP/(2TP+FP+FN)$	[0, 1]	Larger is better	A harmonic mean of the precision and recall
ACC	${\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$	[0, 1]	Larger is better	The percentage of correct decisions made by the algorithm
ARI52–54	${\displaystyle \frac{\mathit{RI}-\text{Expected(RI)}}{max(\mathit{RI})-\text{Expected(RI)}}}$	[0, 1]	Larger is better	A similarity measure between two clusterings
${\text{AMI}}_{max}$ 55	${\displaystyle \frac{MI-E(MI)}{mean(H(L_1),H(L_2))-E(MI)}}$	[0, 1]	Larger is better	The agreement of two independent label assignment strategies on the same dataset when the real ground truth is not known.

Sen.: sensitivity; Spec.: specificity; TP: true positive; TN: true negative; FP: false positive; FN: false negative; ACC: accuracy score; ARI: adjusted rand index; AMI: adjusted mutual information.

Extracting the featured PDF for images

To extract the PDF feature of an image, there are two primary approaches, including parametric (generally extracting data according to the Gaussian distribution) and non-parametric (operating the Kernel function)⁵⁶. Methodologically, the assumed image size is $m\times n$. This paper is extracting this image into a PDF by, first, rescaling into [0, 1], straightening the pixels into a $1\times (m\times n)$ vector, and finally estimating into a Gaussian distribution or a kernel function whose bandwidth is chosen by^57–60.

• The Gaussian estimator given by

  $$\begin{array}{c}\hfill f(\mathbf{x})=\frac{1}{\sqrt{2\pi }\sigma }exp\left(\frac{{(\mathbf{x}-\mu )}^2}{2{\sigma }^2}\right),\end{array}$$ 2

  where, $\mu$ and ${\sigma }^2$ the mean and variance of pixel vector.
• The kernel estimator given by^61,62

  $$\begin{array}{c}\hfill f_h(\mathbf{x})=\frac{1}{\mathit{nh}}\sum _{i=1}^{n}K\left(\frac{\mathbf{x}-{\mathbf{x}}_i}{h}\right),\end{array}$$ 3

  where, analogously, n is the sample size, $K(·)$ is the kernel smoothing function satisfy that $K(·)\le 0$ and ${\int }_{{\mathbb{R}}^d}K(x)\text{d}x=1$⁶³, h is the bandwidth or smooth parameter, and ${\mathbf{x}}_i$ is the pixel values.

In summary, we extract a representative PDF for an image of size $m\times n$ based on the following steps.

1. Convert to grayscale

   The image is converted into grayscale and extracted pixels to get the matrix of size $m\times n$.

2. Stretch into column vector

   The obtained matrix is stretched into a vector of size $1\times (m\times n)$.

3. Estimate the PDF

   The received vector next estimated by (2) or (3). This result PDF is a representation of the image.

The proposed algorithm

A brief review of PCF

The possibilistic partition space

Clustering an n-unlabelled PDFs set $\mathcal{F}=\{f_1,\dots ,f_n\}$ is partitioning of $\mathcal{F}$ into a prior number of clusters $1<c<n-1$ clusters such that each subgroup represents natural substructure in $\mathcal{F}$. Let $\mathrm{\Theta }=\{{\theta }_1,{\theta }_2,\dots ,{\theta }_m\}$ be a set of m-PDFs that will be used for the representing (or prototypes) of cluster i formed in $\mathcal{F}$. Moreover, let $\mu :\mathcal{F}\to [0,1]$ be the degree of representativity or typicality of datum $f_j$ with the i-th cluster denoted by $c_i$⁶⁴. Then $\mathbf{U}={\mu }_i(f_j)=[{\mu }_{\mathit{ij}}],i=1,\dots ,c,j=1,\dots ,n$ be an $c\times n$ matrix whose (i, j) element stands for the so-called degree of compatibility. The partitions matrix generated by validity and noise cluster approaches ${\mu }_{\mathit{ij}}$ satisfies the three conditions^14,15:

$$\begin{array}{cc}& {\mu }_{\mathit{ij}}\in [0,1],\forall i=1,\dots ,n,j=1,\dots ,c,\hfill \end{array}$$ C1

$$\begin{array}{cc}& 0<\sum _{j=1}^n{\mu }_{\mathit{ij}},\forall i=1,\dots ,c,\text{and},\hfill \end{array}$$ C2

$$\begin{array}{cc}& \sum _{i=1}^c{\mu }_{\mathit{ij}}\le 1,\forall j=1,\dots ,n.\hfill \end{array}$$ C3

The (C1) are interpreted as the membership degree of j-th PDF to cluster $c_i$ relative to all other clusters. For the (C2), this is a requirement that no cluster, represented as clusters of $\mathcal{F}$, is empty. Thus, we partition the clusters into no less than c. Conditions (C1) and (C2) are the same as FCM, but the third is different. Here, releasing the condition equal to 1, and PCM says that the sum of ${\mu }_{\mathit{ij}}$ less than equal to 1, meaning that an element can belong to different clusters without requiring the sum of the dependence degrees to be equal to 1. (C3) deals with data that are unclear or indistinguishable from belonging to a cluster in the FCM. Let ${\mathbf{U}}_f$ denote the set of all the matrices of size $c\times n$, satisfying the constraints (C1), (C2) and (C3). The set ${\mathbf{U}}_f$ is referred to as the possibilistic partition space.

Possibilistic clustering for PDF

The objective function of the PCF is defined as follows¹⁴.

$$\begin{array}{cc}\hfill \text{Minimize}:& J_{\mathit{PCM}}(\mathrm{\Theta },\mathbf{U};\mathcal{F},c,m,\eta )=\sum _{i=1}^c\sum _{j=1}^n{\mu }_{\mathit{ij}}^m{\Vert {\theta }_i-f_j\Vert }_1^2+\sum _{i=1}^c{\eta }_j\sum _{j=1}^n{(1-{\mu }_{\mathit{ij}})}^m,\hfill \\ \hfill S.t.:& (C1),(C2),(C3).\hfill \end{array}$$ 4

The parameter m ($1<m<\infty$) is a weighting exponent called fuzzifier. When $m\to 1$, the membership function becomes harder; moreover, $m\to \infty$, the memberships are maximally fuzzy. The $\Vert {\theta }_i-f_j{\Vert }^2$ is squared ${\mathcal{L}}_1$-distance between datum j-th PDFs f and i-th prototype $\theta$. The coefficients ${\eta }_i$ (called resolution or scale parameter) are positive that computing by fuzzy intra cluster distance

$$\begin{array}{c}\hfill {\eta }_i=\frac{K}{N_i}·\sum _{j=1}^n{\mu }_{\mathit{ij}}^m{\Vert {\theta }_i-f_j\Vert }_1^2,K>0,1\le i\le c,\end{array}$$ 5

where $N_i={\sum }_{j=1}^n{\mu }_{\mathit{ij}}^m$, from an initial probabilistic fuzzy clustering result. Generally, $K=1$ is frequently chosen.

Assumptions

The assumptions of the PCF method in this work are as follows.

1. This method should be applied when the number of bodies of abnormal is small and difficult to detect, as it introduces a new cluster that, if suitable, reduces the accuracy of the proposed algorithm and loses its purpose of the article.

2. Each PDF represents one and only one image. We can extract images based on vector pixels using many different methods (the most typical for the parameter is the Gaussian method, and the non-parameter is the kernel method). However, an image only receives one corresponding representative PDF.

3. The number of predefined clusters in the PCF is equivalent to the number of predefined labels in the image data set.

The stage of proposed algorithm

This section proposes an iterative refinement technique to seek image abnormalities via PCF, which includes three stages. Initially, the images collected are grouped and simultaneously extracted into PDFs. The PCM algorithm then performs a cluster analysis of the PDFs to evaluate the clustering level and cluster centroid. A partition matrix is then filtered through an $\alpha$-cut threshold to detect anomalies in the clustering result matrix. Finally, the evaluation results classify an image as abnormal or not. The proposed method summary is detailed in the following algorithm. Given $\mathcal{D}$ data including n-PDFs on c-cluster ($2\le c\le n$).

• Step 1 Reading the PDFs

  Depending on whether the data is discrete data or images sets, we first convert it into PDFs to obtain CDF data ${\mathcal{D}}_f=\{f_1,f_2,\dots ,,f_n\}$ using the appropriate methods in (2) or (3).

• Step 2 Establishing the PCF

  Next, taking ${\mathcal{D}}_f$ as the extracted PDFs and letting c be the predetermined number of clusters, performing the following steps.

  • Step 2.1 Initialization
    We initialize fix values of c, m, MaxIter, $\epsilon >0$, where $2\le c\le n,1<m<\infty$.
    Then, we randomly initialize ${\mu }_{\mathit{ik}}$ $(i=1,\dots ,c\text{and}k=1,\dots ,n)$ such that (C1) , (C2), and (C3), for all i, k.
    The value of ${\eta }_i$ is then estimated using Eq. (5);
  • Step 2.2 Representation step (Fix the membership degree ${\mu }_{\mathit{ik}}$)
    The possibilistic membership degree ${\mu }_{\mathit{ik}}$ is updated using the Eq. (7);
  • Step 2.3 Allocation step (Fix the prototype ${\theta }_i$)
    The prototype ${\theta }_i$ of the cluster $C_i$ is computed using Eq. (12).
  • Step 2.4 Stopping criterion
    If $\Vert J_{iter+1}-J_{\mathit{iter}}\Vert <\epsilon$ or $iter>MaxIter$, we stop the step, otherwise we update $iter:=iter+1$ and go Step 2.2.

• Step 3 Detecting the possibilistic results

  Finally, the Detection steps filter outliers from data points by selected a threshold $\alpha \in (0,1)$.

At the end of the algorithm, we receive the identifier of each PDF about the cluster to which it belongs and whether it is abnormal.

In Step 2, we obtain the possibilistic-partition matrix $\mathbf{U}=[{\mu }_{\mathit{ij}}]$ result in the last iteration and prototypes $\mathrm{\Theta }={\theta }_i$ of the i-th cluster. The criteria $\epsilon$ is a minimal positive number chosen arbitrarily. The smaller $\epsilon$ is, the more iterations and the computer’s time are taken. Generally, we choose $\epsilon =5\times {10}^{-4}$ or $1\times {10}^{-4}$. The MaxIter is set to be 300. Note that we need to initialize c-centroid randomly in Step 2.1.

In Step 3, the threshold defines an $\alpha$-cut (or $\alpha$-level set) on membership of PCM in Step 2 by the rule that the element detected is abnormal if all the clusters membership value rejected itself through $\alpha$-cut value as the Eq. (6). This discrete function A defined by $\mathcal{C}(\mathrm{\Omega })\to \{0,1\}$ has the value of whether the state of an element is identified as abnormal or not. This function, seeded as the decision-making step and used to recognize the abnormalities, forms as (6). Finally, to build the identical partition matrix, we consider the abnormalities in the image as a new cluster.

$$\begin{array}{c}\hfill A(f_k)=\left\{\begin{array}{cc}1\hfill & u_{\mathit{ik}}<\alpha ,\forall i,\hfill \\ 0\hfill & otherwise.\hfill \end{array}\right)\end{array}$$ 6

The flowchart presented in Fig. 1 illustrates the sequence of steps needed to execute the proposed algorithm for the calculation process.

Figure 1.

Flowchart of the proposed method.

Convergence of the PCF algorithm

The following two theorems confirm the establishment of two formulas for the partition matrix and the updated centroid for PCF.

Theorem 1

(Possibilistic membership degree) Let $\phi :{\mathbf{U}}_f\to \mathbb{R},\phi (\mathbf{U})=J(\mathbf{U},\mathrm{\Theta })$ defined by (4), where $\mathrm{\Theta }$ is fixed. Then ${\mathbf{U}}^{\ast }\in {\mathbf{U}}_f$ is a global minimum solution of $\phi$ if and only if

$$\begin{array}{c}\hfill {\mu }_{\mathit{ij}}=\frac{1}{1+{\left(\frac{\Vert {\theta }_i-f_j{\Vert }^2}{{\eta }_j}\right)}^{1/(m-1)}},1\le i\le c,1\le j\le n.\end{array}$$ 7

The necessary conditions on the prototypes are identical to the corresponding requirements in the FCM and its derivatives. This updated formula is independent of any distance measure.

Proof

Minimization of $\phi$ over ${\mathbf{U}}_f$ is an optimization Kuhn-Tucker problem with $c(n+1)$ constraints (C1), (C2) and n constraints (C3). Over these constraints, the original optimization problem is rewritten as⁶⁵

$$\begin{array}{cc}\hfill \text{Minimize}:& \phi (\mathbf{U}).\hfill \\ \hfill S.t.:& y_{\mathit{ij}}(\mathbf{U})={\mu }_{\mathit{ij}}-1\le 0,\hfill \end{array}$$ 8

$$\begin{array}{cc}& z_{\mathit{ij}}(\mathbf{U})=-{\mu }_{\mathit{ij}}\le 0,\hfill \\ \hfill & {\zeta }_j(\mathbf{U})=-\sum _{i=1}^c{\mu }_{\mathit{ij}}<0,\hfill \\ \hfill & {\varsigma }_j(\mathbf{U})=-\sum _{j=1}^n{\mu }_{\mathit{ij}}<0,\hfill \\ \hfill \text{where}& 1\le i\le c,1\le j\le n.\hfill \end{array}$$ 9

Suppose that ${\mathbf{U}}^{\ast }$ is a minimizer of the above objective function $\phi (\mathbf{U})$. Then it must satisfy the following KKT conditions^66,67.

1. ${\mathbf{U}}^{\ast }$ is feasible, i.e.,

   $$\begin{array}{c}\hfill y_{\mathit{ij}}({\mathbf{U}}^{\ast })\le 0,z_{\mathit{ij}}({\mathbf{U}}^{\ast })\le 0,{\zeta }_j({\mathbf{U}}^{\ast })<0\text{and}{\varsigma }_j({\mathbf{U}}^{\ast })<0;\end{array}$$
2. There exist 2cn non-negative complementary multiplies ${\lambda }_{\mathit{ij}}$ and ${\tau }_{\mathit{ij}}$ such that:

   $$\begin{array}{c}\hfill {\lambda }_{\mathit{ij}}{y}_{\mathit{ij}}({\mathbf{U}}^{\ast })=0,\text{and}{\tau }_{\mathit{ij}}{z}_{\mathit{ij}}({\mathbf{U}}^{\ast })=0;\end{array}$$
3. Stationarity

   $$\begin{array}{c}\hfill \frac{\partial \phi }{\partial {\mu }_{\mathit{ij}}}({\mathbf{U}}^{\ast })+\sum _{i=1}^c\sum _{j=1}^n{\lambda }_{\mathit{ij}}\frac{\partial y_{\mathit{ij}}}{\partial {\mu }_{\mathit{ij}}}({\mathbf{U}}^{\ast })+\sum _{i=1}^c\sum _{j=1}^n{\tau }_{\mathit{ij}}\frac{\partial z_{\mathit{ij}}}{\partial {\mu }_{\mathit{ij}}}({\mathbf{U}}^{\ast })=0.\end{array}$$ 10

Substituting (8) and (9) into 1, 2, 3, the three conditions of KKT being:

$$\begin{array}{cc}& 0\le {\mu }_{\mathit{ij}}^{\ast }\le 1,\hfill \\ \hfill & {\lambda }_{\mathit{ij}}({\mu }_{\mathit{ij}}^{\ast }-1)=0,{\tau }_{\mathit{ij}}{\mu }_{\mathit{ij}}^{\ast }=0,\text{and},\hfill \\ \hfill & m{({\mu }_{\mathit{ij}}^{\ast })}^{m-1}{\Vert {\theta }_i-f_j\Vert }^2-m{\eta }_i{(1-{\mu }_{\mathit{ij}}^{\ast })}^{m-1}+{\lambda }_{\mathit{ij}}-{\tau }_{\mathit{ij}}=0.\hfill \end{array}$$

By contradiction, we can easily prove that all the multipliers ${\lambda }_{\mathit{ij}}$ and ${\tau }_{\mathit{ij}}$ are zero because of the non-zero element m, ${\mathcal{L}}_1$-distance, and $\eta$. Substituting ${\lambda }_{\mathit{ij}}=0$ and ${\tau }_{\mathit{ij}}=0$ into (10), we obtain

$$\begin{array}{c}\hfill \frac{\partial \phi }{\partial {\mu }_{\mathit{ij}}}({\mathbf{U}}^{\ast })=m{({\mu }_{\mathit{ij}}^{\ast })}^{m-1}{\Vert {\theta }_i-f_j\Vert }^2-m{\eta }_i{(1-{\mu }_{\mathit{ij}}^{\ast })}^{m-1}=0,\end{array}$$

which established (7). It is clear that ${\mathbf{U}}^{\ast }>0$ for all i, j; thus it is a feasible solution satisfying (i). The necessity is proved.

To show the sufficiency, we examine ${\mathbf{H}}_{\phi }(\mathbf{U})$, the $(cn\times cn)$ Hessian matrix of $\phi$ evaluated at $\mathbf{U}\in {\mathbf{U}}_f$. It is easy to deduce that

$$\begin{array}{c}\hfill \frac{{\partial }^2\psi }{\partial {\theta }_t\partial {\theta }_s}=\left\{\begin{array}{cc}{\xi }_{\mathit{ij}},\hfill & \text{for any}t=s,\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}\right)\end{array}$$

where ${\xi }_{\mathit{ij}}=m(m-1){({\mu }_{\mathit{ij}})}^{m-2}{\Vert {\theta }_i-f_j\Vert }^2+m(m-1){\eta }_i{(1-{\mu }_{\mathit{ij}})}^{m-2}$. Since we assume $m>1$ and ${\mathcal{L}}_1$-distance is non-zero in this section, consequently ${\mathbf{H}}_{\phi }(\mathbf{U})$ is a diagonal matrix with all the diagonal element ${\xi }_{\mathit{ij}}$ positive, i.e., a positive definite matrix. Therefore, ${\mu }_{\mathit{tj}}$ calculated by (7) is the solution to the relaxed optimization problem under consideration. $\square$

Next, we fix ${\mathbf{U}}^{\ast }\in {\mathbf{U}}_f$ and consider the minimization of $J(\mathbf{U},\mathrm{\Theta })$ with respect to $\mathrm{\Theta }$.

Definition 2

(Prototype of clusters) Set the given number of clusters to c, the prototype of cluster i is a non-negative PDF defined as

$$\begin{array}{c}\hfill {\theta }_i=\frac{{\sum }_{j=1}^n{({\mu }_{\mathit{ij}})}^m{f}_j}{{\sum }_{j=1}^n{({\mu }_{\mathit{ij}})}^m},1\le i\le c.\end{array}$$ 11

Theorem 2

Let $\psi :{\mathbb{R}}^{\mathit{cn}}\to \mathbb{R},\psi (\mathrm{\Theta })=J(\mathbf{U},\mathrm{\Theta })$ is squared-distance around prototype calculated when $\mathbf{U}$ is constant ($\mathbf{U}\in {\mathbf{U}}_f$). Then, ${\mathrm{\Theta }}^{\ast }$ is a strict global minimum solution of $\psi$ if and only if ${\mathrm{\Theta }}^{\ast }=G(\mathbf{U})$ formed by (12).

$$\begin{array}{c}\hfill {\theta }_i=\frac{{\sum }_{j=1}^n{({\mu }_{\mathit{ij}})}^m{f}_j}{{\sum }_{j=1}^n{({\mu }_{\mathit{ij}})}^m},1\le i\le c.\end{array}$$ 12

Proof

For each $\mathbf{x}$, examining the Hessian matrix, it is easy to deduce that ${\mathbf{H}}_{\psi }(\mathrm{\Theta })$ is a diagonal matrix after differentiating the objective function given by (4) with respect to ${\theta }_i$ defined by

$$\begin{array}{c}\hfill \frac{{\partial }^2\psi }{\partial {\theta }_t\partial {\theta }_s}=\left\{\begin{array}{cc}2{\sum }_{j=1}^n{({\mu }_{\mathit{tj}})}^m,\hfill & \text{for any}t=s\hfill \\ 0,\hfill & \text{othewise}\hfill \end{array}\right).\end{array}$$

Since $\mathbf{U}\in {\mathbf{U}}_f$ we have ${\sum }_j{\mu }_{\mathit{ij}}>0$ which follows that ${\sum }_j{\mu }_{\mathit{ij}}^m>0$ for any i. That implies that ${\mathbf{H}}_{\psi }(\mathrm{\Theta })$ is a positive definite matrix for all $\mathrm{\Theta }$. Hence, $\psi (\mathrm{\Theta })$ is a strict convex function. As a result, ${\mathrm{\Theta }}^{\ast }$ is a strict global minimum solution if and only if $\frac{\partial \psi }{\partial {\theta }_i}({\mathrm{\Theta }}^{\ast })=2{\sum }_{j=1}^n{({\mu }_{\mathit{ij}})}^m(f_j-{\theta }_i^{\ast })=0$ which are equivalent (12). $\square$

Numerical examples

Simulation strategy

In this subsection, the proposed PCF in “The proposed algorithm” section is employed to solve numerous clustering problems. The outline of the simulation strategy is briefly presented as follows. Firstly, the details of each data set will be shown in Table 2. Subsequently, some adequate parameters are recommended later based on the results obtained from the investigations. Thirdly, in this case, a validation of the fuzzy-based algorithms becomes classifying abnormal/no abnormal with criteria shown in Table 1 is designed to measure the effectiveness.

Table 2.

Detail of quantities and noises of example datasets.

Dataset	Object	Type of PDF	Quantity	Num of abnormal(s)
Benchmark	PDFs	Univariable Gaussian distribution	40	5
COIL20	Gray images ($128\times 128$)	Univariable Gaussian distribution	300	12
Face	Color images ($180\times 200$)	Kernel parameters	126	6

We compare the proposed algorithm to the k-means, the Fuzzy Clustering, and the Self-Update Clustering. They are algorithms established explicitly for the CDF problem^24,38. The two algorithms, k-means and Fuzzy Clustering, simulating from CDE as^9,10 through CDF, are partition cluster analysis algorithms that must set a priori number of clusters. On the contrary, Self-Update is an automatic cluster analysis that determines the appropriate ones.

It should be noted that all algorithms are performed over 30 independent times to guarantee the stability of the algorithm, and results are taken as the average value plus the standard deviation. Further, statistics tests are also performed to validate whether the difference is significant. Finally, some primary conclusions are drawn based on the deduced results. Besides, it is worth noticing that all the problems are implemented on numerical calculation software 2023b-version Matlab on an Intel$_{}^{\circledR }$ Core$_{}^{\text{TM}}$ i5-11400H @ 2.70GHz with 16.0 GB main memory. Those programs have supported effectively solving the complex computation of the numerical examples in Section 4. possibilistic clustering, probability density functions, unsupervised learning, detecting abnormal, image extraction

Data sets

In this work, we tackle the subject from an academic research standpoint. In order to explore the behaviors of the proposed algorithm in standard problems, simulation data are first generated (Example 1). The PCF is then expanded to real-world image applications (Example 2 and Example 3). It is also observed that the amount and complexity of sample data sets rapidly rise from Benchmark examples to real-world applications. Furthermore, the data set has an explicit distribution; the PDF of that data set is computed using the precise method (see Example 1). However, suppose the data set does not yet know the distribution, as in cases 2 and 3. In that case, the Gaussian and kernel methods are used to estimate the PDFs for quantity and noise details; see Table 2.

Example 1.

In the first example, we sketch  40 univariate normal PDFs with the same variances and different means. We review this example to illustrate the theoretical results and to test the suitability of the proposed theories and algorithms. Specifically, the $\mathcal{F}$ set consists of 35 univariate normal PDFs are considered $\mathcal{F}={\{f_i\}}_{i=\overline{1,35}}$ with ${\sigma }_i=0.008$ for all ${\mu }_i$, $i=1,\dots ,35$ having formula

$$\begin{array}{cc}\hfill 24& \text{PDFs with}{\mu }_{C_{1i}}=0.246+0.004i,i=1,\dots ,24,\hfill \\ \hfill 11& \text{PDFs with}{\mu }_{C_{2i}}=0.625+0.005i.i=25,\dots ,35.\hfill \end{array}$$

We, next, add simulation outliers $\mathbf{N}$ consists of 5 univariate normal PDFs $\mathcal{N}={\{f_i\}}_{i=\overline{1,5}}$ having same variances of $\mathcal{F}$ and expectation value ${\mu }_{{\mathcal{N}}_i}=\{0.10,0.55,0.56,0.57,0.90\}$. Figure 2a shows the Benchmark CDF estimated from these parameters.

Example 2.

Some samples of the COILS data set are taken, as shown in Fig. 5 found at Columbia University Image Library (COIL-20) including 1440 $128\times 128$ grayscale images (20 objects with 72 poses each), with 16384 features (reals in the range $0-1$). The PDFs estimated by (2) from these images are demonstrated in Fig. 6.

Example 3.

The 126 images from the face data set are involved in the facial recognition task downloaded at Facial Images: Faces94, separated into six categories as different identical individuals as introduced by⁶⁸. The face94 data contains images of people of various racial backgrounds, genders, and ages, and we selected a subsample of nine volunteers. After rescaling to grey levels, PDFs estimated by (3) are demonstrated in Fig. 9. This data set is more complicated than the previous two due to the higher overlapping degree between the considered classes.

Figure 2.

The simulation Benchmark CDF clustered by the proposed method (above) and the FCF (below).

Figure 5.

Some sample images of the COILD-20 dataset. (a) The cluster data; (b) abnormal added.

Figure 6.

The estimated PDFs for 4 clusters and the prototypes of these clusters, abnormals are black PDFs.

Figure 9.

The partition matrix of face images clustered by the proposed method.

Results

Example 1

To illustrate the performance of the proposed algorithm, a test was conducted on an artificial synthetic dataset named Benchmark. Figure 2a shows the clustering results of the comparative methods. A comparison was made using the FCM method, and its effects are displayed in Fig. 2b. The partition matrix results of the PCF and FCF methods are also shown in Fig. 3a and b, respectively.

Figure 3.

The partition matrix of Benchmark CDF clustered by the proposed method (above) and FCF (below).

First, applying both cluster analysis methods, PCF and FCF classify PDF elements into two distinct clusters. The cluster analysis results for maximum defuzzification can be observed in Fig. 2a, which shows that the representation PDFs are biased when clustered using FCF. At the same time, PCF does not exhibit the same phenomenon. Cluster representation PDF bias occurs when the density of PDF elements has imbalanced differences. Second, the partition matrix of the proposed method shows that the last five elements seem not to be identified by two other clusters. These elements all have shallow membership degrees in both clusters. In other words, they are abnormal and not accepted in the two groups. Meanwhile, FCF has the behaviour of trying to cluster the analysis of all five elements into two predetermined clusters. This behaviour unintentionally causes itself to be blindsided by unusual values, thus leading to the stereotypical clustering of anomalous PDFs.

Different threshold levels influence the decision that an element is anomalous in the data. Figure 4 investigates the G-mean and F1 scores, which change when the threshold is specified from 0 to 1. Specifically, in the Benchmark data, the decision threshold ranges from 15% to 55% for the best results. However, given other data, the threshold value needs to be tested because it is constructed manually and is unpredictable. This is also a topic that needs further attention.

Figure 4.

The distribution of G-mean (red line) and F1-score (blue one) when linearly investigating alpha values from 0 to 1.

To compare the proposed algorithm, we refer to classical algorithms such as k-means and FCF (formerly fuzzy clustering method for CDE), even with the modern Self-Updated clustering that can auto-determine the number of clusters developed by^24,38. These three methods are all used for analyzing well-known PDFs. Table 3 gives the results showing each method’s evaluation indices ARI, ${\text{AMI}}_{max}$, ACC and F1-score are calculated.

Table 3.

Comparing the results of possibilistic clustering, probability density functions, unsupervised learning, detecting abnormal, image extraction of PCF with the state-of-the-art algorithms in Example 1.

Method	ARI53	${\text{AMI}}_{max}$ 55	ACC	F1-score
k-means ($k=2$)	0.81	0.64	0.91	0.90
Fuzzy clustering ($k=2$)38	0.82	0.64	0.91	0.90
Self-update clustering38	0.98	0.87	0.99	0.99
Proposed method ($c=2$)	1.00	1.00	1.00	1.00

We first see that k-means and Fuzzy Clustering give results with little synchronization between the clustering created by the algorithm’s prediction and the actual clustering at the average ARI level of 0.81 and 0.82, specifically. Similarly, AMI is also low in the two algorithms at only 0.64. This phenomenon can be explained by the fact that these two algorithms are not set to detect abnormalities in PDFs. The given cluster is fixed and determined only to divide the data into separate clusters, leading to low cluster segmentation accuracy and low irregularities (90% overall accuracy). However, Self-Update Clustering is superior and has almost maximum accuracy. Looking specifically, we see that the clusters predicted by this algorithm are five clusters. The three PDFs between the two main clusters are determined to belong to one cluster. Therefore, the clustering results have improved significantly.

The proposed algorithm detects abnormalities and classifies them into a separate cluster. When evaluated, the results of all indexes reach a maximum of 100%, indicating that the PCF model not only accurately classifies but also ensures the balance between precision and recall. These results demonstrate the effectiveness and potential of PCF in practical applications and further research in data clustering (Fig. 5).

Example 2

In this instance, we are applying the proposed algorithm to tackle the challenging problem of adult image recognition in data mining. We estimate the PDFs by Gaussian for each image using the grayscale distribution of image pixels. Figure 6 clearly shows no significant difference between the outliers and the group of extracted images except for the homogeneity between the PDFs that form the clusters.

Example 2 has a clear difference in the PDFs presented in Fig. 6 in terms of the shape of the density function compared to Example 1. Figure 7 is the resulting matrix partition of PCF analysis. Here, the results show us discriminatory differences between clusters of PDFs. The area with almost all values between the clusters is low; we can suspect this is an abnormal element because it does not belong to the known prior clusters.

Figure 7.

The partition matrix of COIL-20 images clustered by the proposed method.

Table 4 comparing the results of the PCF method with the k-means, Fuzzy Clustering, and Self-Update Clustering algorithms has been presented and analyzed. The results continue to demonstrate the superiority of the PCF method. This method achieved the best performance in all evaluation metrics compared to the matching methods, including ARI, ${\text{AMI}}_{max}$, ACC, and F1-score. PCF achieves the highest ARI and ${\text{AMI}}_{max}$ values of 0.99 and 0.98, respectively. This demonstrates the high similarity between the clusterings generated by PCF and the actual clusterings. PCF also achieved the best overall accuracy (ACC) and F1-score of 1.00 and 0.99. In summary, the results demonstrate that the proposed algorithm can significantly enhance data clustering performance and is well-suited for application in data mining, particularly in image recognition.

Table 4.

Comparing the results of PCF with the state-of-the-art algorithms in Example 2.

Method	ARI53	${\text{AMI}}_{max}$ 55	ACC	F1-score
k-means ($k=4$)	0.93	0.92	0.98	0.95
Fuzzy Clustering ($k=4$)38	0.90	0.87	0.97	0.95
Self-Update Clustering38	0.98	0.96	0.99	0.98
Proposed method ($c=4$)	0.99	0.98	1.00	0.99

Example 3

This subsection presents an application of the proposed method in detecting images with abnormal faces. The PDF is estimated for each image based on the way shown in “The proposed algorithm” section, in this case, the Kernel method. The considered data set contains 126 images. There are six normal clusters in the used data set. Each cluster contains 20 images of the same face with different poses. Additionally, six images of new faces are deemed to be anomalies.

The estimated pdfs are illustrated in Fig. 8. The PDFs are characterized by a high degree of overlapping, a complex shape, and a random distribution of anomalies. Results of Fig. 9 show that a partition matrix with a concentrated level of partitioning is still obtained.

Figure 8.

The estimated PDFs for 6 clusters and the bold lines are prototypes of these clusters; Abnormals are black PDFs.

The last example presents the results of recognizing abnormal facial images of different individuals. In Table 5, the accuracy and F1-score of the proposed PCF have reached 100%, the highest among state-of-the-art algorithms. Moreover, compared with well-known and classical algorithms such as k-means and FCF, the proposed algorithm has AMI and ARI indexes reaching a maximum of 100%, showing that the agreement level between two independent label assignment strategies is very high. In addition, we notice an approximation of 100% accuracy (and F1-score: 99%) in the Self-Update Clustering method due to identifying abnormals as separate clusters of one to three PDFs. However, the clusters are automatically divided purposelessly and not controlled synchronously, leading to a total number of resulting clusters of up to 11 different clusters.

Table 5.

Comparing the results of PCF with the state-of-the-art algorithms in Example 3.

Method	ARI53	${\text{AMI}}_{max}$ 55	ACC	F1-score
k-means ($k=6$)	0.94	0.90	0.98	0.95
Fuzzy clustering ($k=6$)38	0.94	0.92	0.98	0.95
Self-update clustering38	0.99	0.95	1.00	0.99
Proposed method ($c=6$)	1.00	1.00	1.00	1.00

These results verify the PCF method’s effectiveness in data clustering and confirm its progress compared to other state-of-the-art methods in the field of clustering. They also provide a solid basis for its usage in practical applications and further research in data clustering.

Table 6 is calculated with binary accuracy evaluation standards with and without anomalies such as F1 score, G-mean, Sensitivity, and Specificity to evaluate the numerical results quantitatively. Threshold values are set at 20% through the three examples. Overview, results are meanly in all three examples: the average sensitivity is over 90% (Sign test⁶⁹, $p=1.00$), but the specificity still reaches a very high level of 100% ($p=1.00$). From there, the median G-mean went to a level of 96%. F1 Score is rated at a high level of 92%. The three examples highly represent density function extraction from images and classification according to PCF.

Table 6.

Quantitative evaluation of the PCF method summarizing three numerical examples.

Num. Ex	F1-score @20%	G-mean @20%	Sen. @20%	Spec. @20%	Time
Benchmark	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	0.03
COIL-20	0.92 [0.39 0.92]	0.96 [0.74, 0.96]	0.92 [0.58, 0.92]	1.00 [0.94, 1.00]	11.61
Faces	1.00 [0.80, 1.00]	1.00 [0.82, 1.00]	1.00 [0.67, 1.00]	1.00 [1.00, 1.00]	227.10

@20%: $\alpha =0.2$, each criterion be median [Q1, Q3] of independent sample 30 times repeat

Discussion

Advantages of the current study

The above three numerical examples demonstrate the algorithm’s evaluations with improvements for abnormal image detection problems. The effectiveness of the proposed algorithm has been demonstrated in both simulation and practical problems, highlighting its potential worth. Some substantial items include selecting $\alpha$-cut and estimating the PDFs. Moreover, the algorithm presented thoroughness and convergence properties proven by Theorem 1 and 2 and the handling of computational problems in Matlab. More specifically, PCF emphasizes its applicability to real-world scenarios, such as detecting damaged machinery, abnormal signals in medical analysis, abnormal signals time series in economics and finance, and so forth^70–72 and especially prediction probabilistic risk analysis of overhead contact lines based on dynamic Bayesian network as A data-driven integrated framework^73,74.

Limitations and future works

This research’s limitation is that the algorithm’s convergence and local optimal must be hybridized or developed. Besides, the outlier detection technique, mainly based on the $\alpha$-cut, is the handwork choice that needs improvement. Furthermore, it should be noted that using PDFs as features to classify images is appropriate for a few specific problems where the distribution of color plays a factoring role. At this point, all attention needs to be paid to the appropriate image extraction method, such as parametric or non-parametric. Consequently, analyzing a problem carefully before implementing the proposed image algorithm is always essential³⁹. This disadvantage prevents a broad application of the proposed method to the image recognition problem⁴⁰.

In the future, other evolutionary approaches can be applied to improve and make a more efficient PCF for data mining with big data. Specifically, the PCF method can be studied semi-supervised or classified without being affected by noise or detecting built-in anomalies. Furthermore, uncertain information such as PDFs allows for the simultaneous assignment of true, false, and indeterminate membership degrees to an element, may also become subject to neutrosophic statistics^75–78 by uncertainty management tools such as Dempster-Shafer evidence theory^79–81 to handling conflicting data^78,82 and multidimensional decision processing⁸³.

Conclusion remarks

Research on CDF clustering to detect outliers has been limited in our experience. This area may need more attention to exploit and fully apply its potential. Based on the ${\mathcal{L}}_1$-distance, this paper modifies PCF in the spirit of PCM, which is then applied to solve the clustering for PDFs. Three numerical examples, including the synthetic data sets and two image datasets, are performed to demonstrate the superiority of the proposed algorithm in evaluating the accuracy of detecting clustering-facing abnormalities. After repeating 30 times, the results show that in all three examples, the G-mean reached 96% or higher, and abnormalities were identified at an F1-score of 92%. Tests for the stability of the results, the Sign test, also showed that the sensitivity and specificity were the same with statistical significance $(p>5\%)$. This proves that these novelties have enabled PCF to solve the clustering for probability density functions for the first time. Almost all of them show the proposed algorithm’s effectiveness on both simulation and practical problems, such as image recognition. This research focuses on improving clustering performance, which is necessary for data mining with uncertain sources.

Author contributions

All authors contributed equally to this work.

Data availability

The datasets analyzed during this study are openly available from the public data on the website and explicitly given in the paper. Columbia University Image Library (COIL-20): https://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php Facial Images (Faces94): https://cmp.felk.cvut.cz/~spacelib/faces/faces94.html The program code has been transferred to the GitHub repository at https://github.com/hungtrannam/Possibilistic-clustering-for-PDF.

Competing interests

The authors declare no competing interests.