Visual defect inspection of touch screens using multi-angle filtering in curvelet domain

Abstract

Touch screens are widely used in smartphones and tablets. These screens exhibit a pattern of directional, regular lines on their surface. The intricate texture of this background, which quickly causes interference, poses a significant challenge in detecting surface defects. Surface defects can be mainly classified into two types: linear and planar. Existing methods cannot effectively detect both types of defects. This study proposes a curvelet transform-based multi-angle filtering method. It can effectively attenuate regular patterns from panel images with textural backgrounds and preserve fine linear and planar defects in the reconstructed image. Curvelet transform is a multi-scale directional transformation that can capture the curved edges of objects well. The filtered curvelet coefficients are then reconstructed into the spatial domain and binarized using a threshold based on the interval estimation skill. The results of the trial show that the suggested approach can precisely locate and identify defects in touch panels. The rate of defect detection (1-β) stands at 93.33 %. The rate of defect misjudgment (α) is at a low of 1.26 %. The correct classification rate (CR) is impressively high at 98.69 %, indicating that the proposed method provides fine-grained segmentation results over existing methods for detecting surface defects on touch panels.

1. Introduction

Capacitive touch panels have the highest market share among different types of touch panels. To meet the high demand, automated processes must produce them in large quantities. However, some common defects can affect the quality of touch panels, such as scratches, cracks, debris, dirt, and water stains [1]. These defects can be caused by improper wiping or handling of the panels, or by insufficient drying after cleaning. Touch panel surface defects can be broadly categorized into two primary kinds given their form: linear defects, which are directional and include examples such as scratches and cracks; and planar defects, which are irregularly shaped and low contrast, encompassing debris, dirt, and water stains.

Capacitive touch panels have complex background patterns that challenge automated defect detection [2]. The patterns have directional properties that affect defect detection. This study explores the inspection of a capacitive touch panel with a structural textured background. The pattern is uniform on the surface, which makes it hard to detect defects. Fig. 1 shows a touch panel sample and a close-up of the texture pattern. Two test images are acquired from the touch panel sample containing a linear flaw and some planar flaws. The surface pattern is complex, and defects that overlap the pattern are complicated to detect.

Fig. 1.

Two test images acquired from a touch panel sample containing (a) a linear flaw and (b) some planar flaws.

Touch panel manufacturers used to rely on manual visual inspection for appearance defects. They would use lighting and flip the touch panel simultaneously. By changing the angle of the touch panel, they could see the defects by reflection and detect surface flaws. However, this visual method had many drawbacks. It could cause fatigue and injury to inspectors or affect their concentration after a long operation. It demanded a lot of focus and attention. Therefore, inspectors were likely to make mistakes in product quality inspection when they were tired. Some manufacturers have tried to use traditional computer vision inspection systems to capture defect ranges, but they faced challenges from environmental interference and sample background patterns [3,4]. These factors not only amplify the complexity of defect detection but also contribute to an increase in false positives and a decrease in detection rates.

This study delves into the challenge of identifying surface defects on capacitive touch panels. These panels are characterized by regular conductive electrodes and wires, which form the backdrop of their design [5]. The transparent glass of these panels often features a variety of textured background patterns, including various conductive electrodes and lines, depending on the design and functional requirements of different manufacturers. The touch panel under our scrutiny boasts a complex pattern design, composed of grid-like lines intersecting at various angles. This intricate design presents a significant challenge in distinguishing defects from the pattern using computer vision techniques, especially when they overlap.

This research aims to develop an automated technique for detecting surface defects on touch panel images with textured backgrounds. The proposed method uses a curvelet transform and multi-angle filtering to diminish the texture pattern and emphasize defect features. It then distinguishes the defects from the background, achieving full automation of defect detection. The curvelet transform, with its sparse representation capability, serves a vital role in fabric defect detection. The method is both efficient and dependable for identifying surface defects in industrial products. Unlike machine learning and deep learning methods, the curvelet filtering approach does not require training data, which can be beneficial when such data are scarce. The curvelet filtering method also has a shorter inference time for general images, making it robust for practical applications of online surface defect detection. Some implementations of curvelet filtering methods allow the adaptive selection of filter parameters to detect defects in various image textures [6]. Given these advantages, the curvelet filtering method proves to be a potent tool for defect detection in various sectors, including manufacturing and textile industries.

The remaining sections of this paper are organized as follows: It begins by reviewing the existing literature on the methods currently employed by optical inspection systems for touch screens. Next, we detail the proposed image processing methods that are designed to detect imperfections and determine their locations on the touch screens. This is followed by a series of tests to assess both the effectiveness and efficiency of the proposed models, drawing comparisons with conventional techniques. Lastly, we summarize our contributions and propose potential directions for future research.

2. Related works

The production process has become more efficient with automation, but it also produces more untested products than the inspection unit can handle. An automated visual inspection system is necessary for the product inspection process to guarantee that the output quality adheres to professional standards [7,8]. This system can enhance the speed of detection, and decrease the errors and expenses associated with manual inspection of the product. If the quality of the products is not maintained while taking large orders, the risk of returns will have to be faced. Chen et al. [2] developed a visual inspection system to identify defects in resistive touch panels (RTP), which integrates mechanical, electrical, and computer vision components, and uses image processing technology to identify defects of this kind of touch panel. The line structure on the surface of RTP is simple and resembles a checkerboard. Capacitive touch panels (CTP) have more variety and complex patterns, and their pattern changes are larger [9].

2.1. Traditional-based methods

Traditional defect detection methods frequently rely on machine vision and image processing techniques. Several image processing methods have been applied to flaw detection on surfaces of optical glass, such as image filtering, binarization, and edge enhancement [3,4,10]. Linear defects of object surfaces with texture have been detected using Fourier transform, wavelet transform, and broadband filtering [11,12]. For the surface of the touch panels, Gaussian smoothing was used to remove image noise, histogram equalization was used to improve the uneven light problem, and two-dimensional frequency transforms were used to magnify the distinction between defects and background [[13], [14], [15], [16]]. The defects were also automatically classified and quantified in severity [17]. The traditional methods often lack sophisticated algorithms and may not be suitable for widespread application. They often have difficulty identifying and classifying defects accurately [[18], [19], [20]].

To segment the touch panel's pattern and defects effectively, we apply curvelet transform as the frequency domain transformation method and combine it with multi-angle filtering to attenuate the background pattern. The testing samples have various directional changes, and the curvelet transform could better capture the image edges' curves and had a strong directional decomposition ability [[21], [22], [23]]. Hence, we choose curvelet transform as the frequency domain transformation method for analyzing sample pattern changes. Other frequency-domain transformation methods include Fourier transform, wavelet transform, etc. These methods convert the time domain signal into the frequency domain, so the frequency content of the signal can be analyzed. In comparison to the time domain filtering method, the frequency domain filtering method efficiently eliminates the baseline drift in the signal and effectively mitigates low-frequency interference components [17,18].

The discrete curvelet transform (DCT) is a skill for representing images or signals using multi-scale and multidirectional basis functions. It addresses specific intriguing phenomena that occur along curved edges in signals of higher dimensions. It generates sub-band coefficients, with each sub-band capable of being processed independently. The distinctions between the curvelet transform and its similar methods are outlined as follows. Wavelets form the foundation for depicting position and spatial frequency. However, these methods are inadequate in accurately depicting highly anisotropic elements and only include a set number of directional elements, regardless of scale [24]. The shearlet transform is a multi-dimensional function utilized for sparse representation. It is capable of effectively extracting directional features, delivering the sparse approximation of edge and contour features, and exhibiting greater sensitivity to the geometric structure of the image [25].

The contourlet transform is composed of bases with varying directions across multiple scales, exhibits a low redundancy rate, and possesses superior boundary representation capabilities. The distinction from the curvelet transform lies in the fact that the degree of direction localization varies with scale [25]. The ridgelet transform employs the radon transform on a distinct overcomplete wavelet pyramid. The curvelet transform employs the ridgelet transform in a synthesis step and integrates a filter bank made up of wavelet filters to carry out the curvelet subbands. The bandlet transform belongs to a family of multiscale geometric transformations that are utilized in image and signal processing. It is used for sparse representation of an image, which is particularly useful in image compression.

The choice among these transforms depends on the specific requirements of the task, taking into account factors such as the nature of the data and the computational resources available. The DCT can preserve the image quality better than the wavelet transform, which can introduce artifacts and distortions due to its isotropic and homogeneous nature [24]. The DCT can also handle images with different orientations and scales better than the wavelet transform, which can lose information and resolution when dealing with anisotropic and heterogeneous images [26]. Alzubi [24] discussed how wavelet, ridgelet, and curvelet transforms can perform multi-resolution analysis for segmentation systems. It indicated the curvelet transform, being an advanced version of wavelet and ridgelet transforms, is effective in capturing curves.

2.2. Deep learning-based methods

Defect detection techniques that utilize Convolutional Neural Networks (CNNs) have been widely applied in various industrial scenarios [[27], [28], [29], [30]]. These CNN-based approaches significantly mitigate the limitations of traditional methods, such as a low sampling rate and reduced accuracy. CNNs are capable of independently extracting powerful features for defect detection, with little need for prior knowledge regarding the images. They excel at handling complex textures in backgrounds, noise, and fluctuations in lighting conditions. However, the identification of unbalanced samples can result in biased training and impact the model's performance. In many real-world applications, the limited sample size makes it challenging to acquire a substantial number of labeled samples for training. Real-time processing is also a challenge. Although CNN-based methods have made significant progress in this area, there is still room for improvement [31,32].

A Generative Adversarial Network (GAN) is a structure where two opposing networks engage in competition to produce the desired data. GANs have found extensive use in defect detection tasks, owing to their robust data generation capabilities [19,20]. GANs are unsupervised learning algorithms, which means they do not require labeled data for training. This is particularly useful in defect detection where obtaining defect samples can be difficult and unpredictable [33,34]. GANs can generate more realistic and effective data compared to traditional data enhancement methods. However, GAN models are harder to train. They necessitate the ongoing supply of various types of data to verify their accuracy.

A hybrid method, the combination of GANs and Attention Mechanism (AM), can be quite powerful in defect detection tasks [35,36]. The AM can effectively improve the utilization of correlation information, the precise detection of small defect areas, and the clear definition of overlapping boundaries of multiple defects. However, the integrated models encounter the issue of mode collapse, where the training process lacks stability, and the generated images may be confined to a few specific categories or unusual images may emerge. Despite these challenges, the combination of GANs and AM has shown great potential in defect detection and remains a vibrant field of research [37,38].

While defect detection can be achieved using both machine learning and deep learning methods, they vary in their feature extraction approaches and their requirements for data and resources [7,8]. The choice between these methods would depend on the specific requirements of the defect detection task. Machine learning algorithms are significantly dependent on the quality and volume of data. If the training data is not representative of real-world scenarios, the model may perform poorly [[39], [40], [41]]. Machine learning models can sometimes fit the training data too closely, leading to overfitting problems and poor performance on testing data. Furthermore, complex machine learning models can be difficult to interpret and understand, i.e. they lack interpretability. Deep learning models necessitate substantial computational resources, posing a potential constraint in real-time applications. Like machine learning methods, deep learning models can also overfit the training data, which can lead to ineffective generalization when encountering new data. Additionally, the development and fine-tuning of deep learning models typically necessitate domain expertise [7,8].

3. Proposed methods

This study proposes a method to identify surface defects on capacitive touch panels with structural patterns. The method comprises two stages. In the first stage, the curvelet transform is employed for the panel images to produce the curvelet decomposed images. Then, the multi-angle filtering processing is conducted on the detailed images in the curvelet domain based on prior knowledge of the sample's pattern characteristics. Besides, the reverse curvelet transform is conducted to filter out the background patterns and highlight the defects. In the second stage, a binarization skill is used to segment the flaws from the background, with black as the flaws and white as the background. The efficacy of the proposed method is assessed through the outcomes of statistical analysis.

3.1. Image capture

A touch panel sample with dimensions of 0.78 mm (thickness), 227 mm (length), and 147 mm (width) is randomly chosen from the manufacturer's production line. The touch panel has a background texture that requires blue coaxial light sources for illumination. The sample is positioned on an examination platform and captured using a high-power lens for localized regions. The acquired images show the texture and defects of the touch screen. Fig. 2 shows a schematic and photo of the image capture device setup and image capture device layout of the test panel. We also control the environmental illumination to get digital images with suitable intensity.

Fig. 2.

Experimental hardware setup for image acquisition: a schematic diagram and the corresponding photograph.

3.2. Curvelet transform and decomposed images

To detect surface defects on the touch panel, we need to minimize the interference of the background conductive lines and magnify the contrast of the defects. We use the curvelet filtering method to process the captured images. This method can remove the texture pattern and enhance the defect contrast. Curvelet transform decomposes an image into different detail images that capture the texture information of various directions, and an approximation image that preserves the original appearance of the image. We use a curvelet transform to break down a test image of the capacitive touch panel into several detail images and one approximation image. The advantage of curvelet transform is that it can represent more directional patterns in images, and store them in different detail images. By knowing the direction of the patterns in each detail image, we can filter out the unwanted information more precisely and achieve defect enhancement.

Curvelet transformation implements ridgelet analysis on Radon transform [22]. This method is computationally unique in that it determines coefficients for each scale, orientation, and location. In digital image analysis, the input image is structured as a Cartesian array. Thus, rotation becomes shear and the calculation takes place in the pseudo-polar plane [23]. The curvelet transform, an extension of the wavelet transform, offers superior directionality and reconstruction abilities. Candès et al. proposed an enhanced edition of the curvelet transform termed the fast discrete curve transform (FDCT) [42]. This edition is more efficient, straightforward, and less redundant compared to the initial ridgelet transform.

The discrete curvelet transform (DCT) comprises three major procedures: (1) partitioning the image into dyadic blocks of different sizes, (2) applying a two-dimensional fast Fourier transform (FFT) to each block, and (3) mapping the Fourier coefficients to a polar grid and applying an inverse FFT to gain the curvelet coefficients. The DCT can be implemented using various algorithms, such as the FDCT, the unequally spaced fast Fourier transform (USFFT), or the wrapping-based discrete curvelet transform (WDCT) [42]. The steps of applying the WDCT method in this study are summarized as follows: First, we apply a forward 2-D FFT to an original image g(x, y) of pixel-size M×N and obtain its frequency domain notation G[u,v] as shown,

$$G\left[u,v\right]=\frac{1}{MN}\sum _{x=0}^{M‐1}\sum _{y=0}^{N‐1}g\left(x,y\right)\mathit{exp}\left[-j2\pi \left(\frac{ux}{M}+\frac{vy}{N}\right)\right]\text{.}$$ (1)

Second, we multiply G [u, v] by a set of local window functions ${\tilde{V}}_{j,l}$ that are based on the ideal CT conditions. Each window function corresponds to a scale j and an orientation l, resulting in the product ${\tilde{V}}_{j,l}\left[u,v\right]G\left[u,v\right]$. Third, we wrap the products into rectangles around their origin using the wrapping function W as shown,

$${\tilde{G}}_{j,l}\left[u,v\right]=W\left({\tilde{V}}_{j,l}G\right)\left[u,v\right]\text{.}$$ (2)

This produces the wrapped coefficients ${\tilde{G}}_{j,l}\left[u,v\right]$. Fourth, we perform a reverse 2D FFT to each wrapped coefficient and obtain the inversed FFT image $g^{\prime }\left(x,y\right)$ as shown,

$$g^{\prime }\left(x,y\right)=\sum _{u=0}^{M‐1}\sum _{v=0}^{N‐1}{\tilde{G}}_{j,l}\left[u,v\right]\mathit{exp}\left[j2\pi \left(\frac{ux}{M}+\frac{vy}{N}\right)\right]\text{.}$$ (3)

Fifth, we extract the discrete curvelet coefficients $c^D\left(j,l,k\right)$ from $g^{\prime }\left(x,y\right)$ by multiplying it with digital curvelet waveforms ${\varphi }_{j,l,k}^D$ as shown,

$$c^D\left(j,l,k\right)=\sum _{0\le x,y<m,n}{g}^{\prime }\left(x,y\right){\varphi }_{j,l,k}^D\left[x,y\right]\text{.}$$ (4)

The coefficients ${\mathit{c}}^{\mathit{D}}\left(\mathit{j},\mathit{l},\mathit{k}\right)$ represent the scale, orientation, and position parameters of the curvelets.

The DCT decomposes an image into different scales and angles, and each scale has a different number of angles. The order of the DCT decomposition refers to how many scales and angles are used in the transform. Fig. 3 reveals the initial image and the decomposed images after applying the curvelet transformations in different orders. The differences of various orders of DCT decompositions are mainly in the number of scales and angles, and the resolution and sparsity of the coefficients. The higher the order, the more scales and angles are used, and the higher the resolution and sparsity of the coefficients. Sparsity means that most of the coefficients are zero or close to zero, which implies that the image can be expressed by some notable coefficients. However, higher-order decompositions also require more computation time and memory. Depending on the application, one may choose a lower or higher order of DCT decomposition to balance between accuracy and efficiency.

Fig. 3.

(a) The initial image and the decomposed images after the (b) 1st-order, (c) 2nd-order, (d) 3rd-order, and (e) 4th-order of the curvelet transformation applied.

Fig. 4(a) shows a sample image with lines at 15-degree intervals. We apply the 4th-order curvelet transformation to this image and obtain a decomposed image. The decomposed image consists of a central approximation image (low-frequency region), an outermost black square circle of detail images (high-frequency region), and middle black square circles of detail images (middle-frequency region). In this example, the decomposition number is 16, and the decomposed image after the 4th-order curvelet transformation is marked in the low, middle, and high-frequency regions as shown in Fig. 4(b). The numbers of detail images from the inner square circle to the outermost square circle are 16, 32, and 32 respectively. The numbered outermost detail images in the curvelet domain are revealed in Fig. 4(c).

Fig. 4.

(A) A sample image, (b) the decomposed image after the 4th-order curvelet transformation marked in the low-, middle-, and high-frequency regions, and (c) the numbered outermost detail images in the curvelet domain.

The curvelet domain allows us to decompose the texture in the spatial domain into detail images at different locations. The detail images of each square circle in the curvelet domain have paired texture information. For instance, the 32 detail images in the outer square circle of the 4th-order curvelet decomposed image have the same texture information in the same direction every 16 detail images, as shown by the standard deviation features of a line chart in Fig. 5(a). The first 16 detail images have identical standard deviation values to the next 16 detail images. To select the texture information to be deleted or retained, we need to know the position of the texture in each direction in the detail image in the curvelet domain. Since each square circle has a different number of detail images, we first test the parts marked in red in Fig. 5(b). We delete 32 detailed images in pairs individually and perform inverse curvelet transformation to observe the texture changes. For example, if we delete the 2nd and 18th detail images, which correspond to the filtering locations in Fig. 5(b), we can see that they contain 30° angle direction textures in the curvelet domain. After filtering and reconstructing the curvelet decomposed image, the texture pattern in the 30° direction in the spatial domain image will be attenuated, as revealed in Fig. 5(c).

Fig. 5.

(A) A line chart of the standard deviations of 32 detailed images, (b) the filtered detail images in the curvelet domain, and (c) the reconstructed image indicating the corresponding line has been attenuated.

A test image with colored lines at 15-degree intervals in Fig. 6(a) is used to illustrate the angle correspondence between the initial image in the spatial domain and the detail images in the curvelet domain. We delete each detail image one by one and observe how each one affects the texture orientation in the spatial domain. To facilitate our discussion, we label the detail images with numbers and show their corresponding texture positions and angles in Fig. 6(b).

Fig. 6.

Angle correspondence between the spatial-domain original image and curvelet-domain detail images: (a) a testing image with colored lines at 15-degree intervals, (b) the corresponding detail images with various angles and colors after the 4th-order curvelet transform decompositions. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

3.3. Relationship between decomposition numbers of curvelet detail images and texture directions

The curvelet transformation process allows us to choose how many detail images to decompose. Decomposing more detail images in the curvelet domain helps us separate the texture information in the spatial domain more clearly. For example, if we decompose 180 detail images, we can store the texture information in each direction of 360° in separate detail images with a 1° interval. Fig. 7(a) shows that this method decomposes the texture information finely. However, this also increases the processing time and the storage space of the curvelet conversion. On the other hand, if we decompose too few detail images, we may have texture information from different angles mixed in the same detail image, as Fig. 7(b) illustrates. This makes it harder to detect the defects without removing them precisely.

Fig. 7.

The detail images in the curvelet domain and the number of decompositions of the detail images in each square circle (a) the innermost detail images are decomposed into 180 images, (b) the innermost detail images are decomposed into 8 images.

To evaluate the impact of the curvelet transform's decomposition number and the texture pattern's angle interval on defect detection performance, we conduct experiments with three angle intervals (10°, 15°, 30°) and six decomposition numbers (8, 12, 16, 20, 24, and 28) for different sample textures. We then analyze the relationship between the angle interval in the spatial domain and the decomposition number in the curvelet domain. We also remove the texture information at the same location for each combination of angle interval and decomposition number, and perform curvelet reconstruction.

The pattern's angle interval affects the decomposition of detail images and the preservation of patterns in other directions. If the interval of the pattern angle is 10° and the number of decomposed detail images is 20, we can reduce the erroneous deletion of patterns in other directions. If the interval of the pattern angle is 30° and the number of decomposed images is 8, we will have a better decomposition efficiency. When the textures in the image are successfully decomposed into other detail images, curvelet domain filtering can reduce the number of detail images that need to be removed. Table 1 shows the suitable decomposition numbers of the curvelet detail images for central line patterns with three-angle intervals.

Table 1.

The suitable numbers of detail image decompositions in the curvelet domain for texture patterns at three-angle intervals.

Angle intervals	10°	15°	30°
Suitable numbers of decompositions	20	16	8

The touch panel pattern used in this study is shown in Fig. 8(a). The angles in the texture pattern are 39°, −39°, 90°, and 0° respectively. Therefore, choosing a decomposition angle interval of 10° will give a better description of the patterns in each direction. In this study, the number of decompositions of detail images for this sample image is set to 20. Fig. 8(b) shows the execution result of converting to the 4th order and decomposing into 20 detail images. If the patterns of each angle can be successfully decomposed into individual detail images, then it can avoid deleting too many real defects by mistake.

Fig. 8.

(a) The angles of the texture pattern of the touch panel image and (b) the curvelet domain image after converting the 4th-order curvelet transformation and decomposing 20 detailed images.

3.4. Curvelet multi-angle filtering and filtered image reconstruction

This research focuses on detecting surface flaws of touch panels by removing the regular background texture and preserving the defects. A curvelet transform converts an original image into a set of detail images for each square circle, where the background texture information is decomposed. By knowing which detail image contains the background texture in the curvelet domain, we can filter it out precisely and enhance the defect contrast.

The experiment uses the following parameter settings: the texture angle is 10° and the number of decomposed detail images is 20. An image is broken down into one approximation image and three square circles of detail images, with 1, 20, 40, and 40 images from the innermost to the outermost square circles. The second circle has half the number of decompositions as the third and fourth circles. When the number of decompositions is 20 (i), the circles above the second circle have 40 (2i) decompositions. This reduces the number of second-circle detail images and makes them contain texture information from multiple angles, which can cause defects to be mistakenly deleted.

To filter the detail images in different ways, we focus on the third and fourth circles and remove the 1st, 5th, 11th, and 15th detail images in the second circle. These four detail images also have texture information from four angles. Table 2 shows the texture distribution of the first 20 detail images in the third and fourth circles. The texture information in the 21st to 40th detail images is symmetrical to that of the first 20 detail images.

Table 2.

The directions of the texture pattern stored in the first 20 detail images of the outermost two square circles (the third and fourth circles) in the 4th-order curvelet decomposed image.

No.1	No.2	No.3	No.4	No.5	No.6	No.7	No.8	No.9	No.10
40°	30°	20°	10°	0°	−0°	−10°	−20°	−30°	−40°
No.11	No.12	No.13	No.14	No.15	No.16	No.17	No.18	No.19	No.20
−50°	−60°	−70°	−80°	−90°	90°	80°	70°	60°	50°

We describe a multi-angle filtering test in the curvelet domain for a touch panel image with defects. The curvelet decomposed images without and with partial detail images filtered in various angles show the filtered locations in red in Fig. 9. Fig. 9(b) filters out the ±40°, ±90°, and ±0° directions and then returns to the spatial domain. Fig. 9(c) filters out the ±30°, ±40°, ±90°, and ±0° directions and then returns to the spatial domain as well. Fig. 9(b1) and Fig. 9(c1) are the reconstructed images of the two filtering methods and will be compared and evaluated for the performance of filtering on defect detection. The curvelet transformation is a reversible process that decomposes an image in the spatial domain into one approximation image and several detail images. To reconstruct a filtered image from the curvelet domain, a reverse transformation is needed after applying any filtering operation.

Fig. 9.

Curvelet decomposed images without and with partial detail images filtered in various angles (in (a), (b), (c)), and their corresponding reconstructed images in the spatial domain (in (a1), (b1), (c1)).

3.5. Defect separation from the reconstructed background

This study proposes a method for identifying surface defects on touch panels that have regular textures. The direct binarization of the image fails to eliminate the background texture smoothly and affects the defect separation. To conquer this problem, we suggest using curvelet transform and frequency domain filtering to magnify the distinction between the defects and the background. Then, a simple thresholding skill is employed to segment the flaws from the background. The pixels with grayscale values below the threshold are considered defects and assigned a value of 0, while the remaining pixels are deemed as background and assigned a value of 255.

The curvelet filtering technique is a multiscale transform that can isolate different scales and orientations of image features. Adopting this technique to a test image can magnify the distinction between the defects and the background pattern, which has regular lines and textures. Fig. 10(a) represents the initial image, and Fig. 10(d) exhibits the reconstructed image after applying curvelet filtering. As we can see, the background patterns are suppressed and the flaws are more prominent. This makes it easier to use a simple classification method to detect and separate them from the background.

Fig. 10.

Defect detection procedures ((a) to (e)) of a touch panel image using curvelet multi-angle filtering approach.

4. Experiments and results

This study uses a personal computer (CPU: Intel® Core 2 3.00 GHz, 2 GB RAM, OS: win7), a 5-megapixel B/W CCD, a 35 mm lens, and a coaxial light source to capture images of capacitive touch panels. The images are taken from a local sample area of 9.5 × 6.0 cm, as shown in Fig. 1. The coaxial light is placed directly above the sample. Fig. 2 shows the lighting method. The images are cropped to 256 × 256 pixels for further processing. This study uses Matlab R2009 software to write a program that can apply curvelet transform and filtering to the images and segment the defects using binarization. The performance evaluation results are presented. Fig. 11 shows the layout of the graphical user interface of the conducted vision system.

Fig. 11.

The layout of the user interface for the developed inspection system.

This study aims to assess the defect detection performance by comparing the identified defects with manual evaluations. The performance indicators are normal area misjudgment rate α, defect area detection rate (1-β), and accurate classification rate (CR). The parameter setting results based on these indicators are shown. The parameters include the order of curvelet transform decompositions, the filtering positions, and the binarization threshold. These parameters affect the performance evaluation indicators. To find the suitable parameter settings, this study adopts a two-stage approach: first, it uses a small sample of images to test different combinations of parameters and select the best ones; then, it validates the selected parameters on a large sample of images. Table 3 gives the sample sizes for the small and large trials.

Table 3.

The sample sizes of testing images used in this research experiments.

Experiments with various sample sizes	Testing images			Total amount
	Defect-free	with linear defects	with planar defects
Small sample size	20	20	20	60
Large sample size	60	60	60	180

4.1. Selections of decomposition orders of curvelet transform for flaw detection

This study applies the curvelet transform to the sample images with different orders of decomposition. We choose the number of orders based on the direction of the sample pattern. The suitable order number is explored to achieve the highest detection result in terms of effectiveness and efficiency. For this batch of test samples, we used i = 20, which generated 40 detail images for each of the third-order and fourth-order detail images. These detail images captured the pattern information more effectively. We only processed the third-order and fourth-order detail images, since they matched the number and direction of the sample pattern. We set the binarization parameter k to 2 and filtered out the regions with 40°, 90°, and 0° angles. We tested the performance of the curvelet transform with the 3rd, 4th, and 5th orders of decompositions. Table 4 shows the evaluation results, and Fig. 12 shows the defect detection images for different decomposition orders. For a fixed binarization parameter and the same filtering positions, the 4th-order curvelet transform produces better filtering results than other orders. Hence, we use the 4th-order curvelet transform in our subsequent experiments.

Table 4.

Various decomposition orders of curvelet transformation applied to flaw detection and corresponding performance evaluation results.

Decomposition orders	Third-order	Fourth-order	Fifth-order
(1-β) %	92.88	92.87	90.79
α %	2.19	1.58	1.61
CR %	97.75	98.36	98.32

Fig. 12.

Resulting images of defect detection by various decomposition orders of curvelet transformation.

4.2. Selections of filtering location types

We conduct further experiments with the curvelet transform decomposition order fixed at 4 and the binarization threshold k set to 2 for choosing the filtering locations in the curvelet domain. We apply four methods from Table 5 to filter the detail images of the third and fourth circles in the curvelet domain and summarize the detection results. We find that method 2 (removing 30°, 40°, 90°, and 0°) for filtering is suitable to improve the defect detection performance for panel images.

Table 5.

Four types of filtering locations and corresponding filtering angles in curvelet domain applied to flaw detection and their performance evaluation results.

Types of filtering angles	(1). Three angles (±40°、90°、0°)	(2). Four angles (±30°、40°、90°、0°)	(3). Five angles (±20°、30°、40°、90°、0°)	(4). Six angles (±10°、20°、30°、40°、90°、0°)
(1-β) %	92.87	92.26	91.62	89.05
α %	1.66	1.32	1.27	1.26
CR %	98.30	98.61	98.65	98.65

4.3. Selections of the threshold for image binarization

The curvelet transform, a multi-scale directional method, filters out the background texture and amplifies image defects. The suitable binarization parameter k for separating defects from textures depends on the order and the filtering angles of the curvelet decomposition. In this study, we fix the order to 4 and the filtering angles to 30°, 40°, 90°, and 0°. We then vary k and evaluate the performance of defect detection using ROC curves and CR values. The trial results of these parameter changes are presented in Table 6 and Fig. 13. We find that k = 2 gives the best performance, with (1-β), α and CR being 92.26 %, 1.32 %, and 98.61 %, respectively. Fig. 14 shows the defect detection images with different k values.

Table 6.

Effect of changes in threshold values on the image binarization of flaw detection.

Thresholds	k = 1.3	k = 1.5	k = 1.7	k = 2	k = 2.3	k = 2.5
(1-β) %	98.35	97.36	95.56	92.26	86.87	82.94
α %	7.66	4.77	2.84	1.32	0.63	0.40
CR %	92.34	95.22	97.12	98.61	99.25	99.45

Fig. 13.

ROC curve of performance evaluation in defect detection using various binarization threshold k values.

Fig. 14.

Resulting images of defect detection with various binarization threshold values k.

This study introduces a method for identifying surface defects on touch panels, utilizing small-sample experiments and choosing the appropriate parameter settings. We use curvelet transform to decompose the images of the touch panels into different directions and scales, and then delete some detail images to magnify the distinction between the defects and the background. We also apply binarization to segment the defects from the background. We find that the appropriate parameter setting combination: the decomposition order of curvelet transformation is 4, the detail images of 30°, 40°, 90°, and 0° direction patterns are filtered, and the binarization threshold value k is 2. We verify our method on large-sample experiments and summarize the detection results in Table 7.

Table 7.

The appropriate parameter setting combination is used for the detection results of large sample experiments.

Decomposition order	Filtering angles	Binarization threshold k	(1-β) %	α %	CR %
4	±30°、40°、90°、0°	2	93.33	1.26	98.69

4.4. Result comparisons of flaw detection by the conventional methods and suggested approach

This work uses a curvelet transform with multi-angle filtering properties to attenuate the background texture of a test image and enhance the visibility of defects within the image. To assess the efficacy of this approach, this study will contrast it with some common spatial domain techniques, such as Otsu and Iterative methods, as well as other frequency domain techniques. One of the frequency domain techniques that will be used for comparison is the Fourier transform with broadband filtering, which was proposed by Lin and Tsai [11] for touch panel defect detection.

This study evaluates the performance of different defect detection methods on 180 touch panel sample images, consisting of 60 images with linear defects, 60 images with planar defects, and 60 normal images. Fig. 15 illustrates some of these images. As seen in Fig. 15, the detection methods of Otsu [43] and Iterative [44] can detect the defects, but they also preserve the background texture, which leads to many false positives. Among the frequency domain approaches, Lin and Tsai [11] applied Fourier transform with broadband filtering to weaken the texture pattern in the background and enhance the defects. However, this method cannot fully detect many planar defects, as shown in Fig. 15. Table 8 compares the results of this study with other methods for the 154 testing images. The suggested method reaches a large defect detection rate of 93.33 %, a low false positive rate of 1.26 %, and a high accuracy rate of 98.68 %. The experimental findings illustrate that the suggested approach performs better than other methods for both linear and planar defects, but it is slightly slower than some other methods in terms of processing time.

Fig. 15.

Images of partial defect detection results of large sample trials using spatial and frequency domain approaches.

Table 8.

Comparison table of performance evaluation on defect detection in touch panel images by various methods.

Domains	Spatial domain		Frequency domain
Methods	Otsu method [43]	Iterative method [44]	FT-based filtering method [11]	Proposed method
(1-β) %	99.85	99.83	83.51	93.33
α %	44.11	41.23	3.1	1.26
CR %	56.19	59.05	96.63	98.68
Time (s)	0.03	0.05	1.93	2.28

In this study, we separate the images of linear and planar defects for performance evaluation comparison. The Fourier transform with broadband filtering method [11] mainly targets linear defects for processing. As shown in Table 9, this method achieves higher detection results for linear defects but misses a large number of planar defects. Moreover, the proposed method is proficient in identifying both linear and planar defects.

Table 9.

Detection performance comparison table for linear and planar defects by the FT-based band filtering method and the proposed method.

Defect types	Linear defects		Planar defects
Methods	FT-based band filtering method [11]	Proposed method	FT-based band filtering method [11]	Proposed method
1-β (%)	98.71	97.23	68.31	90.16
α %	3.33	1.72	2.87	1.19
CR %	96.66	98.26	96.61	98.68

4.5. Flaw detection on panel images with the coexistence of linear and planar defects by the proposed approach

Touch panels may have linear and planar defects in the same screen during manufacturing. Most existing methods deal with these defects separately. This study aims to develop a more practical method to handle both types of defects simultaneously and evaluate their performance. We use the same parameter settings as in the previous trials (the 4th-order curvelet transformation; filtering angles: ±30°, ±40°, ±90°, ±0°). We tune the binarization parameter k to achieve good performance evaluation results with our method. Table 10 shows the performance evaluation summary, and Fig. 16 illustrates some defect detection examples. They all reveal that our proposed method can accurately localize linear and planar defects in the same image.

Table 10.

Detection performance comparison table for panel images with the coexistence of linear and planar defects by the proposed method with different binarization threshold values k.

k	2.1	2	1.9
(1-β) %	87.56	89.93	92.30
α %	0.61	0.75	0.95
CR %	99.11	99.03	98.89

Fig. 16.

Some results from the detection of flaws in panel images exhibiting both linear and planar defects (images shown in the first row (a) are original images, and the second (b) and third (c) rows are the detection results by the proposed method and inspectors, respectively).

The experimental results reveal that the proposed approach successfully detects linear and planar defects on touch screens. However, there are some limitations to this research method that can be addressed for further improvement: (1) The proposed approach depends on prior knowledge of the sample's pattern characteristics for multi-angle filtering. For improved detection results, the parameters of the approach need retraining when different texture patterns are encountered; (2) To minimize interference from background textures, adjustments to the filtering locations are necessary when the sample changes or is misaligned during imaging; (3) Currently, the method cannot detect surface defects on touch panels with varying background patterns simultaneously. However, this can be achieved if the curvelet coefficient is utilized to distinguish the detailed image positions related to the sample background pattern information.

5. Concluding remarks

This research aims to devise an automated technique for identifying surface defects in touch panel images with textured backgrounds. The proposed method, using curvelet transform and multi-angle filtering, handles various defects and enhances them against the background texture. The method applies the curvelet transform to the panel images, conducts multi-angle filtering processing on the detail images, and inverts the filtered curvelet coefficients back to the spatial domain. This results in a smoother background and sharper defects. An interval estimation skill then computes a binary threshold value, which segments the image into defects and background areas. The method exhibits exceptional detection performance. The rate of defect detection (1-β) stands at 93.33 %. The rate of defect misjudgment (α) is at a low of 1.26 %. The correct classification rate (CR) is impressively high at 98.69 %, indicating the proposed method provides fine-grained segmentation results over existing methods for detecting surface defects on touch panels. It is not only effective but also efficient in simultaneously detecting planar and linear defects in capacitive touch panels.

This paper presents a preliminary application of the curvelet transform with multi-angle filtering for surface flaw identification in touch panels. However, the test samples have some limitations that can be addressed and investigated further. Future research directions include: (1) The test sample has a non-directional bridge connector, which may be mistaken for a surface defect by this research method. To avoid this error, the method should be able to distinguish between bridge connectors and defects automatically, thereby lowering the false positive rate of normal areas; (2) Defects may occur on both sides of a transparent touch panel, and these defects can be captured by the camera when imaging. For more precise quality assurance of the panel, it is essential to determine on which side of the panel the defect is located; (3) Examine existing methods to determine the most efficient and effective strategy for the proposed application.

Data available statement

The data will be made available on request.

CRediT authorship contribution statement

Hong-Dar Lin: Writing – review & editing, Writing – original draft, Supervision, Project administration, Methodology, Funding acquisition, Formal analysis, Conceptualization. Jen-Miao Li: Validation, Software, Methodology, Data curation. Chou-Hsien Lin: Writing – original draft, Visualization, Validation, Investigation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.