Visual defect obfuscation based self-supervised anomaly detection

Abstract

Due to scarcity of anomaly situations in the early manufacturing stage, an unsupervised anomaly detection (UAD) approach is widely adopted which only uses normal samples for training. This approach is based on the assumption that the trained UAD model will accurately reconstruct normal patterns but struggles with unseen anomalies. To enhance the UAD performance, reconstruction-by-inpainting based methods have recently been investigated, especially on the masking strategy of suspected defective regions. However, there are still issues to overcome: (1) time-consuming inference due to multiple masking, (2) output inconsistency by random masking, and (3) inaccurate reconstruction of normal patterns for large masked areas. Motivated by this, this study proposes a novel reconstruction-by-inpainting method, dubbed Excision And Recovery (EAR), that features single deterministic masking based on the ImageNet pre-trained DINO-ViT and visual obfuscation for hint-providing. Experimental results on the MVTec AD dataset show that deterministic masking by pre-trained attention effectively cuts out suspected defective regions and resolves the aforementioned issues 1 and 2. Also, hint-providing by mosaicing proves to enhance the performance than emptying those regions by binary masking, thereby overcomes issue 3. The proposed approach achieves a high performance without any change of the model structure. Promising results are shown through laboratory tests with public industrial datasets. To suggest EAR be possibly adopted in various industries as a practically deployable solution, future steps include evaluating its applicability in relevant manufacturing environments.

In the manufacturing industry, ensuring product quality is of paramount importance, which can be automated by machine vision systems^1,2. Machine vision systems for defective product detection can be implemented with machine learning or deep learning-based models. However, a significant challenge arises when confronted with the scarcity of anomaly situations, leading to an imbalanced dataset during the early stages of manufacturing. In such cases, training of an anomaly detection (AD) model under full supervision becomes practically unfeasible.

Recognizing this predicament, the manufacturing industry has increasingly turned to an unsupervised anomaly detection (UAD) approach. The data imbalance problem is eased simply by UAD because it only exploits prevalent normal samples for the training stage and does not require any defective samples. The rationale behind this approach hinges on the idea that a well-trained UAD model excels in the accurate reconstruction of normal patterns but falters when trying to reconstruct unseen anomalous patterns. This is referred to as contained generalization ability³.

Recent years have witnessed a large amount of research efforts aimed at enhancing the UAD performance by exploring novel neural network (NN) structures and innovative training strategies. Those can be divided into two main categories: (1) employing an additional module to existing NNs such as generative adversarial networks (GAN)^7–10 or memory module^11–14 and (2) changing of the training strategy to online knowledge distillation^14–16 or utilization of synthetic data^17–20. Those methods successfully improve the performance by refining widely adopted mainstream NNs such as U-Net²¹. However, amid the pursuit of ever-more sophisticated techniques to achieve better performance for specific benchmark datasets, their solutions have common limitation of increase of computational expense by employing large-scale deep NNs.

To avoid the above situations, the reconstruction-by-inpainting approach^6,22–28 have been investigated to improve the UAD performance without increasing the scale of the NN structure to use. This approach fundamentally prevents accurate reconstruction of unseen anomalous patterns by making them not visible through masking. However, there still remain the following problems to address: 1) inference latency due to multiple masking or progressive inpainting strategy^6,25–29, 2) output inconsistency by random masking^6,22–24, and 3) inaccurate reconstruction of normal patterns due to a large mask ratio^30,31.

To solve these issues, this study introduces a novel approach to enhance the UAD performance based on single deterministic masking. The proposed method, dubbed Excision And Recovery (EAR), features attention-based visual defect obfuscation. That is, suspected defective regions are obfuscated by mosaicing as shown in Fig. 1. EAR leverages the ImageNet⁴ pre-trained DINO-ViT⁵ that is known to have the ability to emphasize class-specific spatial features. This property is exploited to highlight saliency regions within a given image and excise suspected anomalous regions for inpainting. This deterministic single masking strategy allows fast processing and secures the output reliability. Also, the problem of inaccurate reconstruction of the normal pattern due to large masked region is eased by the mosaic hint which is provided in the masked regions. For this, the proper mosaic scale is estimated for the defective region by leveraging the ratio of principal curvatures of Hessian matrix, which was also used in scale-invariant feature transformation (SIFT)³² to compute the degree of edge response. Thereby, EAR achieves the UAD performance enhancement while it does not change the NN structure at all. The details of the above design components will be described in the “Methods” section.

Figure 1.

An overview of EAR. EAR takes the reconstruction-by-inpainting approach and is characterized by single deterministic masking and visual obfuscation of masked regions for hint-providing. The orange box shows the excision process by exploiting the ImageNet⁴ pre-trained DINO-ViT⁵ to mask suspected defective regions. To promote the reconstruction of the region into a normal form, visually obfuscated information by mosaicing is provided as a hint. At this time, mosaic scale, m, is estimated from the saliency region of the given product image to provide a proper hint. Mosaic obfuscation is performed by average pooling with $m\times m$ pixels of image and upscaling it into the original scale. The blue box shows the recovery process that reconstructs the corrupted region in $I^{\prime }$ into $\widehat{I}$. Abnormality is decided based on a maximum value of $D(I,\widehat{I})$ as shown in the gray box. Note that $D(I,\widehat{I})$ is a distance map calculated by multi-scale gradient magnitude similarity (MSGMS)⁶.

Experimental results with the public industrial visual inspection dataset, MVTec AD³³, demonstrate that EAR further enhances the UAD performance compared to the same or similar scale of NNs. Visual reconstruction results in Figures 4 and 5 indicate that EAR has desirable contained generalization ability³ for the UAD task. That is, suspected defective regions that are visually obfuscated are reconstructed accurately when the input pattern is in the seen normal category, and that the reconstruction is inaccurate when the input includes unseen anomalous patterns.

Figure 4.

Visual comparison of the results when disabling each design component of EAR: visual obfuscation by mosaicing and saliency masking. The EAR variants to confirm the aforementioned effect are the following: 1) EAR is a full model that activates all the components. 2) ${\text{EAR}}_{w/oobf}$ does not provide visual obfuscation-based hints on masked regions. 3) ${\text{EAR}}_{w/oattn}$ disables the masking component which exploits the ImageNet⁴ pre-trained DINO-ViT⁵. A full model of EAR accurately reconstructs normal regions within a defective sample, marked in the yellow box. In contrast, anomalous regions, marked in the red box, are transformed into a normal form and yield a large reconstruction error. ${\text{EAR}}_{w/oobf}$ and ${\text{EAR}}_{w/oattn}$ cases show inaccurate inpainting compared to EAR. Best viewed in color.

Figure 5.

Visual comparison when the input corruption methods, including masking and mosaicing, vary. To visualize the results of RIAD⁶, we have implemented and tested it for each subtask. The RIAD⁶ results show just one masking case among multiple disjoint masks and a cumulated error map for multiple inferences. They show large edge errors overall. For the red spot pattern at the pill, ${\text{EAR}}_{w/oobf}$ shows a mistake in inpainting, and ${\text{EAR}}_{w/oattn}$ produces scattered errors all over the region. EAR shows the accurate reconstruction of normal patterns by saliency masking and hint-providing by visual obfuscation. Best viewed in color.

Overall, the contributions of this study are summarized as follows:

• The proposed pre-trained spatial attention-based single deterministic masking method has advanced the state-of-the-art methods in the reconstruction-by-inpainting approach for UAD, securing both higher throughput and output reliability.

• The proposed hint-providing strategy by visual obfuscation on masked regions further enhances the UAD performance with the proposed mosaic scale estimation method.

The remainder of this paper is organized as follows. The “Related Works” section reviews existing literatures and recent advancements related to this study, highlighting the gaps and limitations that the proposed approach aims to address. In the “Methods” section, the proposed methods, including saliency mask generation and the visual obfuscation-based hint-providing method, are presented in detail. The “Experiments” section presents the experimental settings and results. The “Discussion” section provides an in-depth analysis of the results, interpreting the findings in the context of the research questions. The main findings and contributions of the paper are summarized in the “Conclusion” section.

Related works

In the manufacturing industry, product quality assurance is automated with machine vision systems^1,2. For this, non-AI methods by analyzing datasets and building statistical models³⁴, or AI-based online AD methods^35,36 can be considered. Furthermore, an UAD method can be adopted by considering the scarcity of abnormal situations in an early manufacturing stage³. Among them, the reconstruction-by-inpainting approach⁶ is covered more specifically, which effectively improves the UAD without changing the NN structure. Here, this section briefly reviews related works on UAD techniques and reconstruction-by-inpainting techniques.

Simple but powerful UAD models

There have been efforts to enhance the UAD performance based on widely known NNs, such as auto-encoder (AE) or U-Net²¹, without changing much of their structure. Among AE variants are, MS-CAM³⁷ presents a multi-scale channel attention module with an adversarial learning strategy. GANomaly⁷ adopts feature distance loss to perform better normal pattern reconstruction. SCADN³⁸ performs multi-scale striped masking before feeding input to their NN. In cases of U-Net²¹ variants, DAAD¹² includes block-wise memory module, and RIAD⁶ proposes the reconstruction-by-inpainting strategy with multiple square patched disjoint masks. These approaches maximize the UAD performance while keeping the scale of the NN relatively small.

An U-Net²¹ structure is also employed in this work and at the same time, a practically deployable solution is pursued that allows NNs to operate properly in industrial environments as a way for edge computing.

Reconstruction-by-inpainting methods

UAD based on reconstruction-by-inpainting is an effective self-supervision technique for representation learning to prevent an UAD model from accurately reconstructing unseen anomalous patterns^6,22–28. Specifically, methods such as random masking^6,22–24, multiple disjoint masking^6,25, and progressive inpainting from the initial masks^25–29 have been developed.

The common limitation of multiple masking and progressive inpainting is inference latency due to the multiple inferences. In addition, the random masking strategy causes the problem of output inconsistency when applied to the reconstruction-by-inpainting approach. Thus, to develop a practically deployable solution for ensuring real-time defect detection and output reliability, the following should be considered: 1) deterministic mask generation strategy, 2) minimizing the number of masks, and 3) immediate inpainting strategy rather than a progressive inpainting strategy.

To meet the above requirements, this study exploits a pre-trained attention model for deterministic single masking. The deterministic single masking strategy allows real-time processing and, at the same time, secures the output reliability.

Hint-providing strategies for masked regions

Researches report that attention-based saliency masking^39,40 or non-saliency masking^41,42 is more effective and helpful for representation learning. Their intention is to eliminate unnecessary input information for their objective, representation learning or object recognition.

However, since those masking methods will empty all the information in the suspected anomalous regions, accurate reconstruction of normal patterns becomes hard, especially when the masked region is large. To ease this situation, an additional strategy that randomly leaves a few patches within masked saliency areas as hint information for reconstruction can be considered^39,40. This strategy serves to provide initial information for inpainting the masked regions and accurate reconstruction. However, the randomness of their patch hint-providing causes the output inconsistency problem.

This study presents a visual obfuscation-based hint-providing scheme to promote the accurate reconstruction of normal patterns.

Methods

Overview

Due to the class imbalance problem stemming from the scarcity of abnormal situations, this study adopts a self-supervised learning strategy to conduct target representation learning of normal samples. An overall schematic diagram of EAR is shown in Fig. 1. The excision stage is composed of two steps. First, a deterministic single saliency mask, S, is generated from attention map, A, by exploiting the ImageNet⁴ pre-trained DINO-ViT⁵. The resulting saliency mask, S, indicates suspected anomalous regions. Then, a mosaic hint is provided on the masked regions for reconstruction. To provide a proper hint by obfuscation, mosaic scale, m, is estimated from the part of the given product image that correspond to the saliency region. The result of recovery, $\widehat{I}$, will be obtained by feeding $I^{\prime }$ into the U-Net²¹. The magnitude of the reconstruction error, especially the maximum value of multi-scale gradient magnitude similarity (MSGMS)⁶, between I and $\widehat{I}$ is used to determine whether the product is defective or not.

Saliency mask generation

This study aims to develop a real-time and reliable solution by avoiding inference latency and output inconsistency. For this, a deterministic saliency masking strategy is proposed by exploiting a pre-trained self-attention model. Specifically, the ImageNet⁴ pre-trained DINO-ViT⁵ is used in this study which is trained with a self-distillation strategy. First, an input image I is fed into the DINO-ViT⁵ and get an attention map, A, by averaging [CLS] tokens, multi-heads of the last layer. Then, the attention map, A, is binarized by thresholding the upper quartile value $\mu +0.674\sigma$ (Q3)⁴³ of pixel-wise attention scores to generate a binary saliency mask, S. Referring to the probable error⁴⁴, upper Q3 values is regarded as suspected anomalous regions. This allows the mask size to be sufficiently large enough to cover suspected anomalous regions while keeping it reasonably small.

S is used to cut out the suspected anomalous regions in normal samples in training, and the UAD model will be optimized to inpaint the empty region. After training, the recovered masked region by the UAD model will be accurately matched when the I does not include any unseen anomalous patterns. However, if the masked region covers anomalous patterns, the UAD model will struggle to recover its original defective form. Therefore, defective products can be effectively detected due to relatively large reconstruction errors.

Obfuscation-based hint for reconstruction

Saliency masking empties the defective information in the suspected defective regions to help transform masked unseen anomalous regions into normal forms. However, not leaving any clues in the masked region could cause inaccurate reconstruction of normal patterns, degrading the UAD performance.

This study proposes a hint-providing strategy with visual obfuscation on masked saliency regions for accurate reconstruction of normal patterns. For visual obfuscation, it adopts mosaicing of proper scale depending on the defect scale. For mosaicing, each single representative value within each square patch of $m\times m$ pixels is created by average pooling. Thus, the mosaic scale is represented by m. Determining a proper mosaic scale is described in the section “Determining mosaic scale”. The average pooled image is upscaled into the original scale with the nearest interpolation and combined with a saliency mask to provide the masked regions with the proper mosaic hint as shown in Fig. 1. When the mosaic method described above is denoted by M, the hint-providing method is expressed as (1). The processed image $I^{\prime }$ will be fed into the UAD model for reconstruction.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill I^{\prime }=M(I)\odot S+I\odot \overline{S}\end{array}\end{array}$$ 1

This mosaicing with the proper mosaic scale makes anomalous regions visually obfuscated to an extent that helps efficient reconstruction of normal patterns, and contains accurate reconstruction of anomalous patterns.

Determining mosaic scale

Depending on the mosaic scale, there is a difference in the information details of provided hints for masked regions. Since the mosaic scale to use is a factor that determines the reconstruction quality, it directly affects the UAD performance. As the optimal scale of the mosaic for each product is not known in advance, estimating the proper mosaic scale is necessary to give the best possible hint.

To construct a mosaic scale estimation model, the optimal mosaic scale $m_{}^{\ast }$ is obtained for each product in the MVTec AD dataset³³ through grid search.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill r=Tr{(H)}^2/Det(H),H={\mathrm{\nabla }}^{2}I\end{array}\end{array}$$ 2

For summarizing the pixel-wise edge response, the saliency region in I is only leveraged and average the top 10% of them. Let us denote this by $r_{10}$. $r_{10}$ could be successfully related to $m_{}^{\ast }$ and their linear relation is shown in Fig. 2. They show a strong correlation. Products with detailed features or rough surfaces give high values of $r_{10}$ while products with relatively smooth surfaces show low values.

Figure 2.

Linear regression model between $r_{10}$ and $m_{}^{\ast }$. $m_{}^{\ast }$ found by grid search is denoted by blue and red circles for $r_{10}$, and their correlation coefficients are -0.939 and -0.497 for 10 object subsets and 5 texture subsets, respectively. The linear function f to estimate $\widehat{m}$ is shown by the green line. $\widehat{m}$ is determined by quantizing $f(r_{10})$ to the nearest power of 2.

The linear function is optimized of the mosaic scale estimation model for each object and texture subset. An estimated mosaic scale, $\widehat{m}$, is determined by quantizing $f(r_{10})$ to the nearest power of 2. For experiments, $\widehat{m}$ will be used for EAR training, and the results using $m_{}^{\ast }$ will also be presented to verify the effectiveness of the proposed mosaic scale estimation method.

Training objectives

A prior study⁶ has shown a satisfactory result to detect various-sized defects by employing MSGMS as in (3). Their training objective also includes ${\mathcal{L}}_2$ (pixel-wise distance) and structural similarity index measure (SSIM)⁴⁷ which are widely used for training of a reconstruction model. EAR also inherit the above for training and anomaly scoring. In MSGMS, multiple scales N is set to 3.

$${\mathcal{L}}_{\mathit{msgms}}(I,\widehat{I})=\sum _{n=1}^N\left(1-\frac{2g(I^n)g({\widehat{I}}^n)+c}{g{(I^n)}^2+g{({\widehat{I}}^n)}^2+c}\right)$$ 3

$${\mathcal{L}}_{\mathit{comb}}={\lambda }_2{\mathcal{L}}_2+{\lambda }_{\mathit{ssim}}{\mathcal{L}}_{\mathit{ssim}}+{\lambda }_{\mathit{msgms}}{\mathcal{L}}_{\mathit{msgms}}$$ 4

Three loss terms ${\mathcal{L}}_2$, ${\mathcal{L}}_{\mathit{ssim}}$, and ${\mathcal{L}}_{\mathit{msgms}}$ are combined by applying weights $\lambda$ as (4). Then, a loss transformation method LAMP³ is applied on (4). LAMP³ is known to enhance the UAD performance by only loss amplification of the training process. In addition, it can be applied to any UAD training process because it does not depend on NN structures or preprocessing methods. The final loss function for training EAR is (5).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{L}}_{\mathit{comb}}^{\mathit{LAMP}}(I,\widehat{I})=-log(1-{\mathcal{L}}_{\mathit{comb}}(I,\widehat{I}))\end{array}\end{array}$$ 5

Experiments

Experimental setup

To evaluate the performance of EAR, the public industrial visual inspection dataset, MVTec AD³³, is used for the experiments. MVTec AD³³ provides a total of 15 subtasks with 10 objects and 5 textures. Each training set for these tasks only provides normal, anomaly-free samples. The test set includes both normal and defective samples.

Implementation details The proposed model simply inherits a well-known U-Net²¹-like structure as a reconstruction model for experiments. Specifically, an U-Net is constructed as in RIAD⁶. The reconstruction model is structured with five convolutional blocks for the encoder and decoder respectively, and the i-th layer in the encoder is concatenated with $(5$ $-i)$-th layer in the decoder. For the encoder, ‘convolution $\to$ batch normalization $\to$ leaky ReLU activation’ is repeated three times, and ‘upsampling $\to$ convolution $\to$ batch normalization $\to$ leaky ReLU activation’ is repeated three times for the decoder. Note that, the stride is set to 2 in the third layer of each encoder block for spatial downscaling. Also, upsampling with scaling factor 2 and the nearest interpolation is applied in the first layer of each decoder block.

To activate EAR, a pre-trained attention model is required. Many variants of pre-trained ViT are publicly available. EAR adopts one of the state-of-the-art models, specifically ViT-S/8. It is provided in the official GitHub repository, dino (https://github.com/facebookresearch/dino), published by Caron et al.⁵. This ViT is known to have the ability to emphasize class-specific spatial features that lead to unsupervised object segmentation. This property is leveraged to emphasize saliency regions within a given image and cut them out for inpainting.

Mosaic scale estimation This work includes an optimal mosaic scale estimation process. To construct a mosaic scale estimation model, the ground truth of optimal mosaic scale $m_{}^{\ast }$ for each product is needed. Those are initially found in the grid search manner. The results of finding the initial ground truth are given in Fig. 3, and the linear regression model for mosaic scale estimation is shown in Fig. 2. A summary of mosaic scale estimation for each product is also given in Table 1. As can be seen in Table 1, $m_{}^{\ast }$ and $\widehat{m}$ match in most objects. However, there are mismatches for the textures. Accordingly, it can be observed that the AD performance for them using $\widehat{m}$ is in general clearly lower than using $m_{}^{\ast }$. The experiments will be conducted with $\widehat{m}$ to train an UAD model for each subtask. The UAD results from $m_{}^{\ast }$ will also be shown for comparison.

Figure 3.

Grid search results of finding the optimal mosaic scale $m_{}^{\ast }$ for hint-providing. Overall, it appears that a larger mosaic scale is advantageous in obtaining a higher AUROC. However, for products with detailed patterns such as capsule, pill, and screw, a moderately small mosaic scale is recommended. The correlation between the visual characteristic of the product and the optimal mosaic scale is shown in Fig. 2.

Table 1.

Summary of optimal mosaic scale $m_{}^{\ast }$ and estimated mosaic scale $\widehat{m}$.

Objects			Textures
Product	$m_{}^{\ast }$	$\widehat{m}$	Product	$m_{}^{\ast }$	$\widehat{m}$
Bottle	32	32	Carpet	64	16
Cable	32	16	Grid	32	64
Capsule	8	8	Leather	64	32
Hazelnut	64	64	Tile	2	8
Metal nut	32	16	Wood	8	2
Pill	4	4
Screw	8	4
Toothbrush	32	32
Transistor	64	64
Zipper	2	4

$m_{}^{\ast }$ is found by grid search, and $\widehat{m}$ is estimated mosaic scale determined by quantizing $f(r_{10})$ to the nearest power of 2. The linear regression function f for each object and texture is shown in Fig. 2.

Training conditions Hyperparameter tuning is performed in all UAD experiments for fair comparison of each model in the best performance condition. The tuned hyperparameters are: 1) kernel size 2) learning rate 3) scheduling method of learning rate. As learning rate scheduling methods, fixed learning rates, learning rate warm-up⁴⁵, and SGDR⁴⁶ are used. The values used as hyperparameters are summarized in Table 2.

Table 2.

Summary of the tuned hyperparameters and their values.

Hyperparameter	Values
Kernel size (k)	3 and 5
Learning rate ($\eta$)	1e−3, 1e−4, and 1e−5
Learning rate scheduling	Fixed, warm-up45 and SGDR46

The optimal combination of hyperparameters is explored in a grid search manner.

Evaluation metric To evaluate the performance of UAD experiments, the area under the receiver operating characteristic curve (AUROC)⁴⁸ is used. The AUROC is measured based on the anomaly scores for each normal and defective sample within the test set. For anomaly scoring, this study adopts the maximum value of MSGMS between the input I and reconstruction-by-inpainting result $\widehat{I}$ of the UAD model which is capable of detecting various sizes of defects⁶. When the MSGMSs of the UAD model for the unseen anomalous patterns are relatively larger compared to the normal pattern cases, AUROC will be close to 1.

Visual comparison of reconstruction

Visual comparisons of reconstruction results are presented to see whether the reconstruction of normal patterns is accurate when the proposed method EAR is applied. First, as can be seen in Fig. 4, it can be checked on the effect of disabling either of the important design components: saliency masking and visual obfuscation by mosaicing. ${\text{EAR}}_{w/oobf}$ disables visual obfuscation for hint-providing. That is, ${\text{EAR}}_{w/oobf}$ empties saliency region without any hint. ${\text{EAR}}_{w/oattn}$ does not utilize the ImageNet⁴ pre-trained DINO-ViT⁵ to cut out suspected anomalous regions. Thus, ${\text{EAR}}_{w/oattn}$ reconstructs the whole image that is obfuscated. The results for each model are shown for the best hyperparameter conditions. ${\text{EAR}}_{w/oobf}$ shows an inpainting mistake on the background region marked with a red box, and ${\text{EAR}}_{w/oattn}$ struggles reconstructing the normal region within the defective sample. The full model, EAR, accurately reconstructs normal regions within a defective sample, marked in the yellow box. In addition, red-boxed anomalous regions are successfully transformed into a normal form without inpainting mistakes. Those reconstruction results confirm that both saliency masking and mosaic obfuscation for hint-providing play an essential role in achieving contained generalization ability³ by complementing each other.

In Fig. 5, visual comparisons of the EAR variants with RIAD⁶ are presented. RIAD⁶ features reconstruction by inpainting with multiple disjoint random masks and provides a cumulated error map. The reconstruction results of RIAD⁶ are just one case from multiple masking. The overall edge error, MSGMS, is large due to a lot of random patch masks being located with the edge of the object in RIAD⁶ case. ${\text{EAR}}_{w/oobf}$ case shows the inaccurate reconstruction of the normal region by a large binary mask. Especially in a defective pill case, binary masking causes confusion as to whether the empty space should be filled with a red dot pattern or a white color. ${\text{EAR}}_{w/oattn}$ shows scattered minute errors all over the region are produced because of the spatial discontinuity of the input image due to the mosaic. In contrast, EAR accurately reconstructs normal patterns by leveraging the hint-providing strategy from mosaic obfuscation; specifically, the logo and digit printings of the capsule. It successfully transforms the scratched white digit printing ‘500’ into a normal form.

Anomaly detection performance

EAR is trained with anomaly-free samples. Then, the AUROC is measured for each subtask with an MSGMS-based anomaly scoring method. The measured performance is summarized in Table 3. As this study proposes a strategy to maximize the performance without changing the NN structure, the performance is compared with recent studies that use NNs of the same or similar scale.

Table 3.

Summary of the AUROC for the MVTec AD dataset³³.

Model	MS-CAM37	GANomaly7	SCADN38	MemAE11	U-Net21	DAAD12	RIAD6	EAR (proposed)
Backbone	AE	AE	AE	AE	U-Net	U-Net	U-Net	U-Net
Additional Module	Att	Dis	Dis	Mem	–	Dis &	–	–
Bottle	0.940	0.892	0.957	0.930	0.863	0.976	0.999	0.997 (0.997)
Cable	0.880	0.732	0.856	0.785	0.636	0.844	0.819	0.853 (0.871)
Capsule	0.850	0.708	0.765	0.735	0.673	0.767	0.884	0.870 (0.870)
Carpet	0.910	0.842	0.504	0.386	0.774	0.866	0.842	0.850 (0.899)
Grid	0.940	0.743	0.983	0.805	0.857	0.957	0.996	0.952 (0.959)
Hazelnut	0.950	0.794	0.833	0.769	0.996	0.921	0.833	0.997 (0.997)
Leather	0.950	0.792	0.659	0.423	0.870	0.862	1.000	1.000 (1.000)
Metal nut	0.690	0.745	0.624	0.654	0.676	0.758	0.885	0.856 (0.876)
Pill	0.890	0.757	0.814	0.717	0.781	0.900	0.838	0.922 (0.922)
Screw	1.000	0.699	0.831	0.257	1.000	0.987	0.845	0.779 (0.886)
Tile	0.800	0.785	0.792	0.718	0.964	0.882	0.987	0.918 (0.965)
Toothbrush	1.000	0.700	0.891	0.967	0.811	0.992	1.000	1.000 (1.000)
Transistor	0.880	0.746	0.863	0.791	0.674	0.876	0.909	0.947 (0.947)
Wood	0.940	0.653	0.968	0.954	0.958	0.982	0.930	0.946 (0.985)
Zipper	0.910	0.834	0.846	0.710	0.750	0.859	0.981	0.949 (0.955)
Average	0.902	0.761	0.812	0.707	0.819	0.895	0.917	0.922 (0.942)

NNs are structured with simple well-known reconstruction backbones AE and U-Net²¹. For EAR, AUROCs are shown for two cases of $\widehat{m}$ and $m_{}^{\ast }$, in $\widehat{m}$ ($m_{}^{\ast }$) form. Abbreviations of attention module, discriminator, and memory module are ‘Att’, ‘Dis’, and ‘Mem’ respectively.

EAR achieves the best performance in hazelnut, pill, and transistor cases compared to other models. The common characteristic of defective samples in these subtasks is surface damage which can be recovered into normal form by EAR. In cases of capsules, screws, and zippers that show sophisticated features, AUROC is relatively low compared to the highest performance. This is because the detailed pattern alignment of screw thread or the zipper teeth by reconstruction may be slightly missed due to saliency masking and visual obfuscation in the suspected anomalous regions. In case of a screw, the mask created by DINO-ViT⁵ tends to cover the entire object. That is, it makes the screw into an entirely obfuscated object. Since the screw object has multiple dense threads, EAR struggles to reconstruct an anomaly-free screw due to the difficulty of identifying the starting point of the threads. This affects the UAD performance on normal cases. In case of the wood texture, a slight UAD performance degradation is observed due to poor mosaic scale estimation. The texture of the wood is composed of mostly small irregular texture segments between some large irregular segments in contrast to the other four texture objects, which leads to a large $r_{1}0$ value that falls within the outlier area than the other texture objects. The simplest linear regression model we opted in this study for computational efficiency does not perfectly estimate $\widehat{m}$ for the outlier in this case, and the UAD performance is slightly degraded.

In conclusion, EAR achieves AUROCs of 0.922 and 0.942 by utilizing $\widehat{m}$ and $m_{}^{\ast }$ respectively. The performance 0.922 with $\widehat{m}$ indicates that EAR with mosaic scale estimation enhances the UAD performance compared to prior state-of-the-art models. Although the proposed method has some challenging cases in providing hints for objects with irregular patterns (wood) or very dense patterns (screw), an overall performance gain is beneficial. We will continue to investigate into obfuscation improvement such as coordinate adaptive obfuscation together with exploration of the tradeoff of mosaic scale estimation accuracy vs. computational efficiency.

Training and inference speed

The measured processing time is summarized in Table 4. RIAD⁶ takes the longest time for both training and inference due to the multiple masking strategy. For reference, the number of masks used in RIAD⁶ is set to 12 as suggested in the original paper. In Table 4, ${\text{EAR}}_{w/oattn}$ shows the fastest speed because it does not involve generating saliency maps through a pre-trained attention model, ImageNet⁴ pre-trained DINO-ViT⁵. ${\text{EAR}}_{w/oobf}$ and EAR are somewhat slower than ${\text{EAR}}_{w/oattn}$ because they generate saliency maps via a pre-trained attention model. They show 2.35 $\times$ and 1.86 $\times$ faster inference than RIAD⁶, respectively. EAR shows an inference speed fast enough for real-time processing with the highest UAD performance.

Table 4.

Processing time for each training and inference.

Model	Training (sec)	Inference (msec)
RIAD6	35,478	366
${\text{EAR}}_{w/oobf}$	3084	156
${\text{EAR}}_{w/oattn}$	3078	37
EAR	3109	197

In inference, ${\text{EAR}}_{w/oattn}$ is the fastest model, also EAR shows sufficiently fast inference speed of is 1.86 $\times$ faster than RIAD⁶.

Ablation study

An ablation study has been conducted to see how deterministic saliency masking and obfuscation by mosaicing for hint-providing affect the UAD performance. In addition, this experiment also checks the effect of applying the knowledge distillation (KD) method, part of SQUID¹⁴, which uses two of the same NNs as teacher and student, respectively, during the training stage.

The results of the ablation study are summarized in Table 5. The 2nd column shows the cases of ${\text{EAR}}_{w/oobf}$. It appears to be difficult to achieve a high performance because binary masking empties all the information in the suspected defective regions, causing inaccurate reconstruction on both normal and anomalous patterns. On the other hand, ${\text{EAR}}_{w/oattn}$, shown in the 3rd column, confirms that the obfuscation by mosaicing achieves better UAD performance compared to RIAD⁶ and ${\text{EAR}}_{w/oobf}$ because of the relatively accurate reconstruction of normal patterns, reducing false positives. Also refer to the visual comparisons of the above cases in Fig. 5.

Table 5.

Summary of the ablation study.

Model	RIAD6	Ablations			EAR (proposed)
Masking	✓(multi)	✓		✓	✓
Hint			✓	✓	✓
KD14				✓
Bottle	0.999	0.995	1.000	0.994 (0.995)	0.997 (0.997)
Cable	0.819	0.795	0.888	0.851 (0.855)	0.853 (0.871)
Capsule	0.884	0.784	0.918	0.869 (0.869)	0.870 (0.870)
Carpet	0.842	0.848	0.718	0.846 (0.880)	0.850 (0.899)
Grid	0.996	0.969	0.963	0.976 (0.976)	0.952 (0.959)
Hazelnut	0.833	0.986	0.996	0.992 (0.996)	0.997 (0.997)
Leather	1.000	1.000	1.000	1.000 (1.000)	1.000 (1.000)
Metal nut	0.885	0.832	0.841	0.868 (0.868)	0.856 (0.876)
Pill	0.838	0.738	0.867	0.870 (0.873)	0.922 (0.922)
Screw	0.845	0.800	0.825	0.776 (0.854)	0.779 (0.886)
Tile	0.987	0.928	0.939	0.956 (0.956)	0.918 (0.965)
Toothbrush	1.000	0.994	1.000	1.000 (1.000)	1.000 (1.000)
Transistor	0.909	0.891	0.943	0.895 (0.933)	0.947 (0.947)
Wood	0.930	0.904	0.945	0.986 (0.995)	0.946 (0.985)
Zipper	0.981	0.900	0.963	0.951 (0.961)	0.949 (0.955)
Average	0.917	0.891	0.920	0.922 (0.934)	0.922 (0.942)

Except for RIAD⁶, all the other masking cases employ a pre-trained attention-based deterministic single saliency masking. The rightmost two columns report the performance of visual defect obfuscation methods when using $\widehat{m}$ and $m_{}^{\ast }$, in $\widehat{m}$ ($m_{}^{\ast }$) form.

This experiment confirms that the UAD performance is further improved when mosaic is used for hint-providing as shown in the last column, which is the case of EAR. Note that, the full model of EAR exploits both hint-providing and pre-trained attention-based saliency masking.

When the KD strategy from SQUID¹⁴ is additionally applied (the 5th column in Table 5), there is almost no change in the performance. Referring to the expensive training cost of KD due to the use of two identical NNs (for each teacher and student), there is no advantage of additionally employing KD for EAR.

Experiment on novel products

Additional experiment is conducted to check whether the linear regression function f obtained from the MVTec AD dataset³³ works properly on another dataset with EAR. This experiment have used KolektorSDD2⁴⁹ which contains surface images from electrical commutators. The models used for comparison are f-AnoGAN⁸, AE-SSIM⁵⁰, and RIAD⁶. f-AnoGAN⁸ is trained by both reconstruction and encoding errors. AE-SSIM⁵⁰ is a case where ${\mathcal{L}}_{\mathit{ssim}}$ is used instead of ${\mathcal{L}}_2$ in AE training. RIAD⁶ features the reconstruction-by-inpainting strategy with multiple square-patched disjoint masks. This experiment additionally compares with DRAEM¹⁷ and SGSF¹⁸. These models employ a larger NN constructed by one AE and U-Net for DRAEM and two U-Nets for SGSF. They are similar in that they adopt a self-supervised learning strategy for defect segmentation with synthetic anomalous samples.

The results are summarized in Table 6. The measured $r_{10}$ of KolektorSDD2⁴⁹ is 2.18. Then, $\widehat{m}$ for KolektorSDD2⁴⁹ is determined as 64 by quantizing $f(r_{10})$ of texture subset to the nearest power of 2. The AUROC of EAR is higher than the other three UAD models, f-AnoGAN⁸, AE-SSIM⁵⁰, and RIAD⁶ which use NNs of the same or similar scale. EAR also achieves a higher AUROC than DRAEM¹⁷ and SGSF¹⁸. For other mosaic scales, the changes in AUROC are observed as shown in Table 7. The best AUROC was achieved when $m=32$ while $m=64$ set by the proposed mosaic scale estimation shows almost equivalent performance. This suggests that proposed mosaic scale estimation method is legitimate.

Table 6.

The UAD performance for KolektorSDD2⁴⁹.

Model	F-AnoGAN8	AE-SSIM50	RIAD6	DRAEM17		SGSF18	EAR (proposed)
Backbone	AE		U-Net	AE+U-Net		U-Net$\times$2	U-Net
Add-module	Dis.	–	–	Synt.	SSPCAB51	Synt.	-
AUROC	0.550	0.789	0.703	0.811	0.834	0.863	0.941

For EAR, AUROC is measured with $\widehat{m}$. Abbreviations of discriminator and synthetic data utilization for training are ‘Dis.’ and ‘Synt.’.

Table 7.

The UAD performance for various mosaic scales.

m	64 ($\widehat{m}$)	32 ($m_{}^{\ast }$)	16	8	4	2
AUROC	0.941	0.945	0.921	0.861	0.777	0.729

When the estimated optimal mosaic scale $\widehat{m}=64$ is applied, it shows almost equivalent performance compared to when the best mosaic scale $m_{}^{\ast }=32$ is applied. The best mosaic scale is found by the grid search.

The reconstruction results of two models, RIAD⁶ and EAR, are shown in Fig. 6. Due to the spatial discontinuity of each square-patched disjoint mask, RIAD⁶ shows large edge errors overall. EAR shows more accurate normal pattern reconstruction while there are spotted errors due to misalignment of small dot patterns in the visually obfuscated regions. The similar results could be seen as shown in cases of MVTec AD³³.

Figure 6.

Visual comparison of RIAD⁶ and EAR for the KolektorSDD2 dataset⁴⁹. This study has reproduced RIAD⁶ and trained it for the visualization. The results of RIAD⁶ show only one masking case among multiple disjoint masks. RIAD⁶ causes cumulated error due to multiple inferences. This could be observed the same in the MVTec AD³³ experiments. EAR shows a relatively accurate reconstruction of normal patterns than RIAD⁶ by exploiting saliency masking and hint-providing by visual obfuscation with mosaic scale estimation. Best viewed in color.

In conclusion, these experimental results suggest that EAR effectively estimates the optimal mosaic scale for a novel product and achieves high UAD performance with computational efficiency. Future research efforts will be made to achieve better reconstruction of detailed information in normal patterns such as the red dot pattern seen earlier in the pill cases of MVTec AD³³ or the rough pattern on the surface in KolektorSDD2⁴⁹.

Discussion

This work is to find a way to achieve a real-time and reliable solution that avoids inference latency and output inconsistency. With the recent interest in recycling pre-trained attention mechanisms for edge computing³⁶ in various manufacturing industries, EAR that features an attention-based deterministic single masking strategy with pre-trained DINO-ViT⁵ is presented. EAR greatly shortens the training and inference time compared to recent state-of-the-art RIAD⁶ and can be employed in a plug-and-play manner. The advancement of masking strategy from random multiple masking⁶ to single deterministic masking deserves attention in the relevant research community. Also, the proposed optimal visual obfuscation by mosaic scale prediction helps achieve the desired contained generalization ability.

However, despite the advantages of EAR’s performance mentioned earlier, future steps should deal with the issue of mask coverage that ImageNet⁴ pre-trained DINO-VIT⁵ might fail to completely cover the suspected defective regions for all cases. To address this issue, there is a need to explore the relationship between industrial datasets and various pre-trained attention models trained on different datasets^5,36,52,53. Another option for pre-trained attention model is WinCLIP⁵⁴ which is an attention model to find anomalous regions by prompt guides designed for zero-shot anomaly detection. Through this investigation and subsequent analysis, further research will be conducted to mitigate the possible missing mask issue when leveraging pre-trained attention for suspected defective masking strategy.

Conclusion

This study proposes a novel self-supervised learning strategy, EAR, to enhance the UAD-purposed reconstruction-by-inpainting model. EAR has effectively exploited the ImageNet⁴ pre-trained DINO-ViT⁵ to generate a deterministic single saliency mask to cut out suspected anomalous regions. EAR also provides the best possible hint for reconstruction by visual obfuscation with the proper mosaic scale estimation. EAR not only serves the reliability of resulting the output via deterministic masking and hint-providing strategy but also achieves fast inference via single masking. Moreover, the UAD performance is enhanced because hint-providing strategy promotes the accurate reconstruction of normal patterns and effective translation of anomalous patterns into a normal form.

In laboratory tests, EAR shows promising results. The proposed method is distinguished from others by enhancing the UAD performance with computational efficiency demonstrating strong performance in laboratory tests. In future steps, further evaluation in practical manufacturing environments will be conducted by considering the necessity of confirming its practical deployability.

Author contributions

Y.P. and J.Y. designed the experiment, analyzed the data, and wrote the manuscript. Y.P. performed the experiments. S.K., M.J.K., and Y.L. reviewed the experimental results and manuscript. H.S.K. conceptualized the study and provided resources for the experiments. J.Y. supervised the entire study and provided critical feedback. All authors have reviewed and provided feedback on the draft manuscript.

Data availability

The MVTec AD dataset is available from the MVTec (https://www.mvtec.com/company/research/datasets/mvtec-ad). Also, the KolektorSDD2 dataset is available from the Visual Cognitive Systems Laboratory (https://www.vicos.si/resources/kolektorsdd2/). Interested researchers can freely access these datasets.

Competing interests

The authors declare no competing interests.