Design of intelligent inspection system for solder paste printing defects based on improved YOLOX

Summary

Aiming at the current SPI (solder paste inspection) system for printing solder paste similar defects detection accuracy is not high, the system intelligence degree is low and so on, design a for the solder paste similar defects and combined with phase modulation profile measurement technique and improve the YOLOX intelligent detection system. The core of the system is the improved YOLOX depth model based on s-mosica and kt-iou algorithms proposed in this paper. The experimental results show that the proposed s-mosica and kt-iou algorithms can effectively improve the detection accuracy of printed solder paste, and when combined with the YOLOX model, the best 90.33% detection accuracy is obtained, which is better than the detection performance of the existing algorithms in the same scenario, and it provides an effective and feasible reference program for the design of the SPI high-precision intelligent detection system.

Graphical abstract

Highlights

• •We design an intelligent detection system for printed solder paste defects
• •The s-mosica and kt-iou algorithms were developed to improve the end-to-end depth model
• •Improved YOLOX algorithm outperforms existing models and adapts to system operation
• •Packaged software package easily extends to other industrial defect detection tasks

Machine learning; Engineering

Introduction

In the SMT process, 70% of the defects originate from incorrect solder paste printing,¹ industrial applications through the SPI timely screening unqualified printed circuit board (PCB) boards for rework, to avoid unqualified products flow to the next process resulting in greater losses. SPI (solder paste inspection) refers to the solder paste inspection system, which is the main method of the current solder printing quality check in the SMT industry. Printing quality inspection method, inspection items include volume, area, height, XY offset, shape, bridging, etc.² In the development and design of SPI, laser (laser triangulation technology) and PMP (phase modulation profile measurement technique) techniques are generally used.³ Due to the low resolution of the laser, single sampling repeatability accuracy is low, sampling in motion external vibration and transmission vibration on the detection of the larger impact, the monochromatic light of the laser on the PCB board color adaptability is weak, and other shortcomings, the current triangulation technology has gradually withdrawn from the SPI industry, the more advanced PMP has been adopted by the SPI industry, the technology is divided into FOV walk-stop type and Scan,⁴ but FOV walk-stop to minimize the impact of vibration during sampling movement, the sampling and movement will be separated resulting in slower detection speed, while Scan scanning is just the opposite, to ensure the efficiency of the same time, but due to the mechanical vibration the detection accuracy is affected. In this paper, we choose the Scan type technique to reconstruct the morphology of printed solder paste and explore advanced defect detection algorithms to overcome the decline in detection accuracy under the premise of ensuring detection speed.

Physical optics-based PMP Scan type solder paste printing defects inspection system greatly improves the efficiency and quality of the inspection; the defect detection process mainly has two ways: (1) through the technician to observe the reconstructed image, according to the experience of the training to judge and analyze the defects; (2) first of all, the production of standard templates, and then the reconstructed image alignment and standard templates are compared with the brush to pick out the defects.⁵ Both methods have shortcomings. The manual visual inspection method will undoubtedly be attached to human subjective factors, resulting in unstable detection accuracy, and with the increase in work intensity and length of time, the efficiency is greatly reduced. The template matching method requires the production of high-precision templates, manually designed features, and feature extraction, and only after the comparison of the defects screened out, the operation process is cumbersome, seriously affecting the efficiency and accuracy of the detection, and poor portability. Based on the aforementioned problem analysis and summary, PMP technology for solder paste printing defects inspection still has shortcomings. The shortcomings lie in the defect identification and localization link method of the obsolete, inefficient, high error, and non-transportability, especially through the PMP technology to obtain the printed solder paste reconstruction image good part and defective part; the differences in the image characteristics of different defects between the defects are not obvious, which increases the difficulty of defects identification. Therefore, the study of novel defect recognition and localization methods is the key to improving the defect detection performance of existing SPI systems.

Currently, recognition methods based on deep convolutional neural networks have been widely used in many fields, which can automatically learn defect features and realize end-to-end judgment prediction,^6,7,8 greatly improving detection efficiency and accuracy. Although deep learning methods have obvious advantages over traditional methods in the target detection task, the effect of simply transplanting them to the printing solder paste defect detection scenario is not ideal. Yoo et al.⁹ proposed a convolutional recurrent reconstruction network for SPI printing solder paste image reconstruction, which can decompose the abnormal patterns generated by the defects of the printing presses from SPI data and reconstruct errors. The method can decompose the abnormal patterns generated by printing machine defects from SPI data, and reconstruct the error detection patterns by self-learning normal data, to improve the accuracy of recognizing defects. Although there is a certain effect of improving the image quality of SPI reconstructed defects by using the convolutional neural network, the idea of the method still stops at improving the quality of the reconstructed image itself, and it cannot improve the recognition difficulty caused by the similarity of the characteristics of the state of the different solder pastes in the reconstructed image. Zheng et al.¹⁰ address the fact that the increase in the number of circuit board types due to increased customization requirements no longer makes it feasible to develop fine-tuned models for each board type, and the authors propose a fast clustering method to select suitable existing models for training the predicted multitasking targets to avoid retraining and fine-tuning in the inference phase. Although this method takes into account the degradation of detection accuracy due to inappropriate detection model selection, the uncorrected model is difficult to adapt to boards with diverse morphologies, and even for boards with very high similarity, solder paste printing is still highly variable; JM Park et al. used a two-stage defect detection network, PointNet, to automatically acquire defective solder paste pattern images from an SPI machine for point cloud feature extraction and then defect detection at two semantic levels (micro-level and macro-level), obtaining a classification accuracy 10.2% higher than that of the convolutional neural network (CNN)-based model. However, PointNet is not suitable for dealing with the relationship between point clouds and local features, which may lead to some information loss and poor performance in dealing with local feature tasks, and the large-scale point cloud data leads to the system’s memory and computational resources are strained, which seriously affects the system stability and work efficiency.

In summary, a single use of PMP technology to detect defects in printed solder paste makes it difficult to meet the needs of the actual quality inspection, or based on deep learning methods for SPI solder paste image quality improvement, model adaptive matching, 3D point cloud data processing, etc., cannot effectively solve the extraction and identification of similar defects in the solder paste features, which at the same time lead to increased algorithmic complexity and reduced system efficiency affecting real time and stability. Given the aforementioned problems, this paper designs an intelligent detection system for similar defects in solder paste printing based on PMP and deep learning methods, which absorbs the excellent reconstruction capability of the original SPI for the details of the defects in the printed solder paste to present the 3D morphology of the solder paste in a perfect and detailed way, and at the same time, uses a deep learning model based on convolutional neural networks to solve the problem of identifying and localizing the targets of similar defects. The new system consists of two main parts; the first part is the 3D reconstruction process of printed solder paste with PMP technology, and the second part is the recognition and localization of defects; the second part is the key to the intelligent system, which is to train the deep learning model through a large number of SPI images generated in the first part, making it possible to accurately recognize and localize highly similar defects in the scanned images of printed solder paste with PMP. Deep learning network solves three basic problems: the first is the acquisition of training data sources; for this problem, we proposed s-mosica data enhancement algorithm to expand the training dataset; s-mosica algorithm absorbs the mosica¹¹ algorithm’s powerful data enhancement capabilities in addition to the elimination of the impact of very small targets, to improve the speed of convergence of the model and the ability to fit; and then to solve the training separation of similar features defects problem, the kt-iou loss function is proposed to better train PMP scanned images with small differences, by adjusting the parameter t to improve the training gradient of low iou samples, so that low iou samples can get more training opportunities, and by adjusting the parameter k penalty term, so that the training process of similar features produces a larger gradient difference, thus making the model.

The main contributions of this work are as follows.

• (1)By researching and analyzing the current solder paste printing defect detection system in industrial applications, and targeting the existing system which mainly adopts traditional image processing methods such as traditional expert knowledge system, feature alignment, and template matching, with low detection accuracy and efficiency, the need to re-design the detection algorithm when replacing the product, and the complexity and cumbersomeness of the system design, we put forward the new intelligent detection system, which combines the phase modulation contour measurement. The new system combines the excellent detail reconstruction characteristics of phase modulation contour measurement technology and the efficient performance of the deep convolutional neural network to automatically extract features for end-to-end detection of the target, to achieve the goal of intelligent, high-precision, high-efficiency, and highly flexible detection of the system.
• (2)Adequate and good target samples are the basis for training excellent deep learning models, but the lack of training datasets is a common problem in the field of deep learning. This paper proposes an s-mosica data enhancement method to improve the problem of scarce defect samples in the PMP scanning dataset of solder paste printing, and at the same time filter out the very small samples that have little significance for training, to accelerate the convergence speed of the model training and to improve the model generalization performance.
• (3)Even if the detection accuracy of PMP technology can reach the micron level, the solder paste printing defect detection using only this technology is still unable to effectively detect the less tin and bridging defects, because their imaging features are very similar. For the aforementioned problem, this paper proposes the kt-iou loss function to train the similar feature samples, to improve the depth of the model to differentiate the performance of the similar features of the depth of the high level of distinction, and to overcome the SPI system PMP scanning images of similar defects difficult to identify and distinguish the problem of improving the accuracy of the detection.

Related work

This section briefly introduces the PMP phase modulation profilometry technique. Phase measuring profilometry (PMP) is widely used because of its high accuracy, good stability, and easy implementation in engineering; this part of the technique is the first part in the proposed framework of solder paste printing defect detection in this paper.¹² PMP uses sinusoidal stripe projection, and phase shift PMP is a 3D measurement method that uses a combination of sinusoidal stripe projection and phase shift techniques. The principle of implementation is shown in Figure 1, where a series of striped images with a fixed phase shift is used to calculate the phase values containing 3D information about the surface of the object to be measured and thus obtain the object height information.¹³ When the object under test is placed on the reference plane, the original flat sine stripe pattern is distorted due to the height change of the object surface, thus the phase of the sine stripe pattern changes. Therefore, the height information of the object is recorded in the modulated sine stripe diagram.¹⁴ The camera acquires the distorted stripe information, then demodulates the phase and completes the reconstruction of the object’s 3D shape, and the reconstructed image is shown in Figure 2.

Figure 1.

Principle diagram of phasor measurement contouring

Figure 2.

PMP scan reconstruction image

The optical hardware used for phase-measurement profilometry mainly consists of a projection module and an imaging module. The optical axis of the imaging module is perpendicular to the reference plane, intersects the reference plane with point O, and the reference plane is parallel to the imaging chip. On the left side is the projection module; the projection module is placed tilted and the angle between the optical axis of the projection module and the imaging module is θ. The projection optical axis intersects the reference plane at the point O. Ep and Ep' denote the entry and exit pupil positions of the projection optical system, respectively, and Ec and Ec' denote the entry and exit pupil positions of the imaging optical system, respectively. EpEc is the baseline distance d, OEc is the working distance lo of the imaging module, and h is the measured object’s height of the measured object.^15,16,17

Figure 2 shows the reconstructed image of the 3D shape of the solder printing after scanning acquired using the PMP phase modulation contour measurement technique. This image shows the complete shape of the solder after printing, and there are differences in the reconstructed images of different printing qualities of the solder paste, and the system determines the quality of the solder paste printing by comparing it with the normal image.

Results

The s-mosica data enhancement algorithm and kt-iou loss function are proposed in the method section, respectively, and firstly, a series of experiments are conducted on the kt-iou loss function with YOLOv3¹⁸ as the benchmark model on the PCB surface defect dataset released by the Human-Machine Interaction Open Laboratory of Peking University to determine the optimal k and t parameter values. Then based on YOLOX¹⁹ combined with s-mosica data enhancement algorithm to do a full ablation study on the PMP scan dataset of solder paste printing defects; and final validation using test samples from the PMP scan dataset.

Kt-iou loss function experiments and analysis

The kt-iou loss function has two parameters that are variable in the definition domain, where the parameter t regulates the transformation function f acting on different iou intervals and the parameter k regulates the intensity of the action function f. The effects of different values of k and t on the loss function are shown in Figure 3.

Figure 3.

Effect of different k and t values on the loss values

(A) Effect of different t values on the loss value when k = 100.

(B) Effect of different k values on the loss value when t = 0.2.

Figure 3 shows that at different values of k and t, the original iou changes its original value under the action of the transform function f, which in turn changes the loss value of the model training, thus directly affecting the change of gradient and indirectly acting on the update of parameters during model training. (Figure 3A) shows that when the fixed parameter k = 100, different t values correspond to the change curve of loss value. 0 < t < 1, when t = 0.2, it means that the transform function f regulates the low iou target frame, and observing the blue curve, we can see that the loss value of the low iou sample has been maintained at a high level, and increasing the gradient of the low iou sample is beneficial to the model training the more difficult to distinguish samples and the high iou sample. The loss value changes roughly linearly, and the loss value changes very little; (Figure 3B) shows the change curve of the loss value corresponding to different k values when fixing t = 0.2. The change of the k value affects the strength of the transformation function f on iou, and the larger the k value is, the greater the strength of f’s action, the steeper the curve is, and the change is more obvious. However, the value of k should not be too large, too large will lead to gradient explosion and model training failure. The effects of different k and t values of model training are verified in the next experiments, and the selection of appropriate k and t values is important for the robustness and stability of the model. Table 1 records the average detection accuracy (mAP) based on the YOLOv3 detection model with different k and t values in the PCB surface defect dataset.

Table 1.

Effect of different k and t values on detection accuracy (mAP) based on YOLOv3 model in PCB surface defect image

t	k
	0.2	0.5	0.8
−10	0	0	0
10	98.71 (+0.21↑)	98.83 (+0.33↑)	97.92 (−0.58↓)
100	99.46 (+0.96↑)	98.94 (+0.44↑)	98.10 (−0.40↓)
1000	0	0	0

YOLOv3 benchmark accuracy is 98.50%.

The experimental data recorded in Table 1 are experiments conducted on the PCB surface defect dataset published by the Human-Computer Interaction Open Laboratory of Peking University, which was selected because the number of targets per defect is relatively balanced and the sample magnitude satisfies the adequate training of the model without negative impact on the model performance due to sample imbalance and too small dataset, and YOLOv3 was chosen as the benchmark model control because YOLOv3 has recognized stability performance and is commonly used as a benchmark model, while kt-iou replaces the computational rules of the iou part of YOLOv3. The detection accuracy of the benchmark model YOLOv3 is 98.50%, and observation of Table 1 shows that the gradient disappears when k < 0 and the model fails; when k > 0, the detection accuracy improves with the increase of k value, but when k is too large, the gradient explodes and the model fails. It is known that 0 < t < 1, defined as t < 0.5 belongs to low iou samples, and t > 0.7 belongs to high iou samples; when t is taken to low iou interval, the model detection accuracy increases, and when t is taken to high iou interval, the model detection accuracy decreases, which means that the transformation function f makes the low iou samples get better training parameters, where k = 100, t = 0.2 the model gets the highest detection accuracy. To verify the more general k and t taking value law, oriented to the main research object of this paper, solder paste printing defects PMP scanning images, to verify the kt-iou loss function on the model performance impact, the experimental parameter settings are consistent throughout. Table 2 shows the detection accuracy results of the YOLOX model tested after training on the solder paste printing defects PMP scan dataset.

Table 2.

Effect of different k and t values on detection accuracy (mAP) based on the YOLOX model in PMP scanning images of solder paste printing defects

t	k
	0.2	0.5	0.8
−10	0	0	0
10	89.28 (+1.30↑)	88.37 (+0.39↑)	86.91 (−1.07↓)
100	89.83 (+1.85↑)	88.76 (+0.78↑)	87.53 (−0.45↓)
1000	0	0	0

YOLOX benchmark accuracy is 87.98%.

The benchmark model YOLOX detection accuracy is 87.98%; observing and analyzing the experimental results in Table 2, it can be seen that the kt-iou loss function, when combined with the YOLOX model, has the same value law on the task of detecting PMP scanning images of printed solder paste defects, and the best detection performance is obtained in the k = 100, t = 0.2 neighborhood, and the value of k < 0 or k value is too large leading to the explosion of the gradient. The kt-iou loss function with high iou values or too small iou values (e.g., 0.1) for t results in poor model performance, so the values of k and t are feasible in the neighborhoods around 100 and 0.2. The other difference is that the kt-iou loss function and the YOLOX modeling framework have a greater impact on the scanned images of printed solder paste defects with PMP, and the accuracy is improved by 1.85% compared to that of the benchmark. The accuracy is improved by 1.85%, which is greater than YOLOv3 in PCB surface defect detection task, because printed solder paste defective PMP scanned images have more similar targets that are difficult to detect, and the kt-iou loss function helps to solve this problem.

Ablation experiments and analysis of s-mosica data enhancement algorithm

In this section, the PMP scan dataset of solder paste printing defects is chosen to verify that the s-mosica data enhancement algorithm has good performance in small sample datasets. The dataset is not only a small sample dataset, but also the training samples are unbalanced. To eliminate the negative impact of data imbalance, YOLOX is selected as the benchmark model because YOLOX contains many widely recognized innovations, which include mosica and focal loss. mosica is good for solving the small sample problem, and focal loss can solve the sample imbalance problem.²⁰ S-mosica is improved based on mosica to address the fact that too small size training samples increase the model training burden and hinder the model from fitting the correct target better. Table 3 records the results of the ablation experiments done on the PMP scan dataset of solder paste printing defects.

Table 3.

Results of ablation experiments with s-mosica data enhancement algorithm

Algorithm	mAP (%)
YOLOX (mosica, focal loss)	87.98
+ s-mosica	88.65 (+ 0.67↑)

The original YOLOX uses the mosica algorithm, and in this experiment, the modified s-mosica algorithm is used instead of the original mosica for ablation experiments.

Table 3 shows that when replacing mosica with s-mosica by changing only a single condition, there is a 0.67% increase in the average detection accuracy, indicating that s-mosica has more powerful data enhancement than the original mosica. The very small targets generated after mosica may be only a few pixels in size or have a severe aspect ratio imbalance, and these very small targets severely hinder the model’s effective training and convergence speed, affecting the detection performance. By removing the very small targets, the model performance is improved and the convergence speed is accelerated. Figure 4 shows the convergence of the two data enhancement algorithms, s-mosica and mosica, during model training.

Figure 4.

Model convergence during mosica and s-mosica training

Figure 4 shows that the model oscillates during training up to 50 and 210 generations, and this phenomenon occurs due to the change in learning rate. The green horizontal line is the change in the loss value when the s-mosica data augmentation algorithm is used. The decline rate is faster than the red solid line. In contrast, the red realization is the convergence of the model when the mosica data augmentation algorithm is used, which indicates that the s-mosica data augmentation algorithm can make the model have a faster convergence rate. In the training to 280 generations, the green line tends to be smooth and the model has reached the convergence state, while the red line still has a decreasing trend, which indicates that the s-mosica data enhancement algorithm speeds up the model training to reach the convergence state and has a higher training efficiency.

I am validating the performance of the s-mosica data augmentation algorithm and kt-iou loss function on the solder paste defect detection task

To verify the performance of the s-mosica data enhancement algorithm and the kt-iou loss function for practical application, taking YOLOX as the basic framework, replacing mosica with s-mosica and the original iou with kt-iou, and combining the research results of the law of k/t value setting, k is set to 100, and t to 0.2, the new model is trained on the PMP scanning image dataset of solder paste printing defects, and the test set is used for practical detection. Train the new model on the solder paste printing defect PMP scanning image dataset, and use the test set for actual detection. The actual detection effect of the SPI intelligent detection system designed in this paper is shown in Figure 5.

Figure 5.

SPI intelligent detection system detection results in the display

Figure 5 shows that when the system IOU is set to 0.5 and the confidence level is set to 0.36, 4 tin fewer defects and 2 bridging defects are detected. The left image in the interface shows the object to be detected, and the right image shows the detection defect localization and category display. The new model and the original YOLOX model are trained separately under the same settings, and the detection results are shown in Figure 6.

Figure 6.

Comparison of the improved YOLOX and original YOLOX detection results

(A) Actual detection results of the upgraded YOLOX (s-mosica + kt-iou).

(B) Detection results of the original YOLOX.

Looking at Figure 6, it is clear that this test image, which contains both under-solder and bridging defects, has a total of six defects. Although all defects do not fall into the category of small targets in terms of size, all solder paste defect images are very similar and the differences are still minute even for images with normal solder paste quality. The introduction of the kt-iou loss function effectively alleviates this difficult judgment situation. (Figure 6A) shows the upgraded YOLOX model detection results, which will successfully detect all defects, (Figure 6B) shows the original YOLOX model detection results, and (Figure 6B) shows the missed detection of the bridging defects at the upper-left corner of the image, which indicates that the kt-iou loss function effectively distinguishes the similar targets, and reduces the probability of missed detection and misclassification. In actual production, defective products should be detected and those with good quality should not be discarded, so detection accuracy and recall should be considered comprehensively, and F1-Score²¹ should be introduced as a comprehensive index to balance the impact of accuracy and recall to evaluate the classifier more comprehensively. Figure 7 shows the F1-Score plots of the upgraded YOLOX (s-mosica+kt-iou) and the original YOLOX.

Figure 7.

F1-Score plots for YOLOX and YOLOX (s-mosica+kt-iou)

(A) F1 values obtained by the original YOLOX model for different defects detection.

(B) F1 values obtained by the improved YOLOX (s-mosica+kt-iou) model for different defects detection.

The horizontal coordinates in Figure 7 are the threshold values, and the F1-Score values are different for different threshold values. (Figure 7A) shows the F1-score values obtained from the original YOLOX model for both tin less and bridging defect tests, and (Figure 7B) shows the F1-score values obtained from the upgraded YOLOX model for both tin less and bridging defect tests. The threshold value F1-Score = 0.5 is usually taken, and it can be found that the F1-Score value of the upgraded YOLOX is 0.05 larger than that of the original YOLOX for the tin less defect test, and 0.13 larger than that of the original YOLOX for the bridging defect test, whereas the larger the F1-Score value is, the more robust the model is, and the better the performance is, indicating that the s-mosica data enhancement algorithm and the kt-iou loss function have produced a positive effect, allowing the detection model to have reliable and stable detection performance on the PMP scanned image defect detection task.

Using the class activation mapping method,²² the detection results of the original YOLOX model and the new version of the YOLOX model with data enhancement of the s-mosica algorithm and the modification of the kt-iou loss function are visualized, so that it is easy to observe which regions of the image the network focuses on and strengthens after the model training. It is also convenient to analyze and compare the degree of attention to the same target before and after the model modification. Using the original YOLOX algorithm and the improved YOLOX (s-mosica+kt-iou) algorithm, the defect detection and feature heatmap analysis are carried out respectively on a test sample image mixed with two tin-poor defects and four bridging defects, and the results are shown in Figure 8.

Figure 8.

Comparison of the YOLOX model and improved YOLOX (s-mosica+kt-iou) model detection performance and thermogram visualization

(A) Original YOLOX defect target detection results.

(B) Original YOLOX model visualization.

(C) Improved YOLOX defect target detection results.

(D) Improved YOLOX model visualization.

Observing Figure 8, it can be seen that in (Figure 8A), the original YOLOX algorithm detection results, there are two bridging defects incorrectly detected as less tin defects, and there are two other bridging defects missed, which shows that the original YOLOX algorithm does not perform well on the task of defect detection of the scanned image of the printed solder paste PMP, and (Figure 8B) also corroborates the original YOLOX algorithm performance, the model not only incorrectly focuses on the defective targets, but also lacks the intensity of attention to the correct targets; compared with the improved YOLOX model by the s-mosica algorithm and kt-iou algorithm proposed in this paper, all the detection results in (Figure 8C) are correct, and all the target features are correctly emphasized by the corresponding network heatmap in (Figure 8D), which indicates that the improved YOLOX algorithm has a better recognition and differentiation ability of similar features of the scanned image of printing solder paste PMP, and the network has a better recognition and differentiation ability. Better recognition and differentiation ability, and the network visualization results prove that the improved model enhances the intensity of the model focusing on the target.

Evaluation of the detection performance of s-mosica and kt-iou in combination with other detection models

Although there are many methods oriented toward PCB solder paste solder quality inspection, as mentioned in the literature review section, traditional image processing techniques, non-contact non-destructive testing techniques, and single deep learning methods have certain limitations, so the comparison of the detection performance of this paper with the other methods will not consider the methods with limitations, but the combination of non-contact laser profiling technique (PMP) and end-to-end deep learning methods can effectively improve the accuracy, efficiency, and intelligence of detecting highly similar defects in printed solder paste; therefore, this section focuses on the comparison of the detection performance of different excellent deep learning methods for the scanned images of solder paste printing PMP. Meanwhile, to assess whether s-mosica and kt-iou also have good combinability and performance enhancement with other detection frameworks, the main deep learning target detection frameworks are selected for comparative study. It should be affirmed that the PMP scanning dataset of printing solder paste defects, which is the target of this paper, was published in Kaggle by Li et al. 2016, and the authors based their defect detection on the Faster-RCNN (FPN) algorithm and set the iou threshold to 0.5, with F1-Score as a metric. For better comparison with the authors’ proposed detection method, the evaluation metrics of this experiment are consistent with the authors. Since this topic is based on deep learning method defect detection, the adaptability and performance of the advanced target detector in PMP scanned images of solder paste printing defects are investigated. Deep learning target detection algorithms are mainly divided into first-order and second-order since the second-order detectors are no longer updated after Faster-RCNN, and the representative of the first-order detectors, the YOLO series, has already launched the YOLOv8 version, which is behind the first-order detection in terms of performance. This experiment mainly investigates the first-order deep learning algorithms in conjunction with the s-mosica and kt-iou algorithms proposed herein in terms of their performance, such as YOLOv5, YOLOX, and the latest YOLOv8. However, the rankings released by the dataset provider are based on the Faster-RCNN framework and are accomplished in conjunction with the FPN network.²³ Hence, a comparison with the Faster-RCNN (FPN) algorithms better observes the excellent performance of the algorithms proposed in this paper. The experimental setup in this section is consistent with the full text and the results are shown in Table 4.

Table 4.

Performance variety of different models combined with s-mosica and kt-iou

Algorithm	F1-Score (score_threshold = 0.5)
Faster-RCNN(FPN)a	0.84
YOLOv5(CIoU loss)	0.79
+s-mosica	0.82 (+0.03↑)
+ kt-iou	0.84 (+0.05↑)
YOLOX (IoU loss)	0.83
+s-mosica	0.86 (+0.03↑)
+kt-iou	0.91 (+0.08↑)
YOLOv8 (CIoU loss+VFL loss)	0.86
+s-mosica	0.88 (+0.02↑)
+kt-iou	0.89 (+0.03↑)

^aindicates the algorithm used by the author of the dataset; score threshold = 0.5 indicates when the threshold is set to 0.5, i.e., the probability is greater than 0.5 for true and less than 0.5 for false.

The experimental results recorded in Table 4 are the weighted average of the F1-Score values for both types of defects, tin less and bridging; ∗ indicates that the best F1-Score test result published by the publisher of the PMP scan image dataset for solder paste printing defects is 0.84. This experiment compares the results in two dimensions: the first dimension is the comparison of different classical excellent deep learning algorithms, and in this dimension YOLOv8 obtains the maximum F1-Score (score_threshold = 0.5) value of 0.86, which indicates that YOLOv8 is best applied without any targeted improvement; the second dimension is the effect of s-mosica data enhancement algorithm and kt-iou loss function on the performance of different deep learning algorithms, respectively, and observes that the performance of different deep learning algorithms is affected by the addition of s-mosica and Through this series of comparative experiments, two conclusions can be illustrated; firstly, during the training process of deep learning algorithms, the s-mosica data enhancement algorithm has great significance in promoting the convergence of model training and the improvement of model generalization in small-scale datasets; secondly, the kt-iou loss function helps the model to distinguish between targets with similar features. By experimental comparison, the YOLOX algorithm trained after s-mosica data enhancement together with the kt-iou loss function exerts more training attention on the difficult-to-distinguish similar samples, and obtains the highest F1-Score (score_threshold = 0.5) value of 0.91 on record. Therefore, in this paper, we use the YOLOX (s-mosica + kt-iou) as the underlying algorithm for the intelligent detection system of printing solder paste defects.

Discussion

In this work, we design an intelligent detection system for similar defects in printed solder paste, which can get rid of the drawbacks of complex system design, low detection accuracy, and poor adaptability brought by the traditional SPI using an expert knowledge system or traditional image processing methods such as feature alignment, template matching, etc., and have better detection accuracy and efficiency for defects with high similarity. The system consists of two main parts, which are based on PMP scanning solder paste imaging and improved YOLOX deep learning network as the intelligent detection algorithm driving the system operation. These two parts work closely together to complete the training and intelligent detection tasks of the system.

S-mosica data enhancement algorithm plays an active role in the training phase of the deep learning network; qualified model training dataset is the basis for deep learning algorithms to have a strong performance, but the acquisition of reliable datasets is a huge challenge at present; Therefore, in the case of a limited dataset to make the model adequately trained, the data enhancement algorithms can effectively improve the model’s generalization performance and robustness; among the many data enhancement algorithms, s-mosica outperforms others. Meanwhile, we also propose the kt-iou loss function to guide the deep network to train more indistinguishable similar samples, to obtain more accurate localization in the target frame regression process, and to improve the detection accuracy of similar targets. By fusing s-mosica and kt-iou with a current representative and excellent detectors, it is found that the best detection performance and stability are obtained when fused with YOLOX.

It should be emphasized that the s-mosica data enhancement algorithm and the kt-iou loss function can be generalized to most deep learning algorithms since data enhancement is one of the important tasks in learning algorithms, and kt-iou improves the regression task based on the basic iou loss, which is also equally effective. Therefore, s-mosica and kt-iou can improve most of the existing deep learning-based target detection models, but the effect may not be too obvious for models without a target frame regression task.

In summary, this paper develops an intelligent detection system for printing solder paste defects based on phase modulation contour measurement technique and an improved YOLOX model, which is fully flexible because the underlying algorithm is a deep network model based on learning; as long as the sample data of the model defects are provided for training, the model can accurately discriminate the different types of defects, which largely improves the system to face the defects. When the engineering application of product replacement and the detection system can be followed by effective training to obtain accurate defect detection capabilities, which has a positive effect on the degree of intelligence to improve the quality inspection of printed solder paste.

Limitations of the study

The current research topic still has limitations, and the best detection accuracy ultimately recorded is only 90.33%, which may be caused by the imbalance of the samples. The detection accuracy of the few tin samples, which account for the majority, is as high as 96.31%, while the detection accuracy of the bridging samples, which account for the minority, is only 84.35%, so effectively solving the sample imbalance problem will help to improve the overall performance of the model and the system. Meanwhile, instead of fusing the s-mosica and kt-iou algorithms with all the deep target detection models and comparing the performance after the training, this paper chooses the YOLO series of algorithms as the main comparison object, including the newly introduced YOLOv8 algorithmic model, to select the improved YOLOX model with the best performance to be used as the underlying algorithm of the system. Further improvement of the system’s performance requires sample balancing and more extensive model comparison validation.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
PCB Surface Defect Dataset	Open Laboratory of Human-Computer Interaction, Peking University	https://www.kaggle.com/datasets/akhatova/pcb-defects
The PCB-AoI Public Dataset	China Telecom and Raisecom Technology	https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi
Software and algorithms
Mosica	Bochkovskiy et al.24	https://github.com/Tianxiaomo/pytorch-YOLOv4
GIoU	Rezatofighi et al.25	https://github.com/diggerdu/Generalized-Intersection-over-Union
DIoU	Zheng et al.26	https://github.com/Zzh-tju/DIoU-pytorch-detectron
CIoU	Zheng et al.27	https://github.com/Zzh-tju/CIoU
Improvement of the YOLOX model improvement section (s-mosica and kt-iou)	This study	https://github.com/jackong180/s-mosica-and-kt-iou
PyTorch	Version 1.11	https://pytorch.org/docs/1.11/
Anaconda	Version 3.0	https://www.anaconda.com/download
Python	Version 3.8.8	https://www.python.org/downloads/release/python-388/

Resource availability

Lead contact

Requests for further information and resources should be directed to the primary contact, Defeng Kong (kongdefeng0916@gmail.com).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• •The PCB surface defects dataset has been published in publicly accessible kaggle repository. Accession numbers are listed in the key resources table. The PCB-AoI public dataset has been published in publicly accessible kaggle repository. Accession numbers are listed in the key resources table. The dataset is publicly accessible.
• •All code associated with this paper can be freely accessed and downloaded via https://github.com/jackong180/s-mosica-and-kt-iou.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Experimental model and study participant details

PCB surface defect dataset

The PCB surface defect dataset²⁸ released by the Open Laboratory for Human-Computer Interaction, Peking University has 693 images with 3–5 defects on each image. The defects include six types: missing hole, mouse bite, open circuit, short, spur, and spurious copper. The defect images of the original dataset are high resolution and for such a small dataset, data enhancement techniques are used before data training. The images were then cropped into 600 × 600 sub-images, resulting in a training set and a test set of 9920 and 2508 images, respectively.

PMP scan datasets for solder paste printing defects

The PMP scan dataset of solder paste printing defects is part of the open-source distributed collaborative AI benchmarking project KubeEdge-Ianvs. It is published by KubeEdge SIG AI members from China Telecom and Raisecom Technology. The dataset collected over 230 pcb scanned images and enhanced them to over 1200 images by rotating them at different angles. There are two types of defects collected in the dataset, namely, tin less and bridging.²⁹ Figure 6 shows a sample of some of the images. (Figure 9A) the blue box area shows the bridge defect, a thin strip linking the solder joint and the chip base surface, (Figure 9B) the red box shows the less tin defect, the defect is mainly in the middle area of the chip pin and solder joint combination.

Figure 9.

Detailed diagram of under-tinning and bridging defects

(A) Blue boxed area shows bridging defects.

(B) The Red box shows in less defects.

Introduction of evaluation metrics

The evaluation indexes in this paper contain AP, mAP, Recall, and F1-score values. Their meanings are respectively as follows:

AP: average correct rate is the result of deriving good or bad detection for each class; mAP: average class AP, i.e., the average of AP of each class; Recall: the quotient of the number of images that the classifier judges as positive class and the true class is also positive class and the number of images where the true class is positive class, which measures the ability of a classifier to find all the positive classes; F1-score: F1 is the summed average of precision and recall, and the larger the F1-score, the higher the quality of the model.

Experimental platform and parameter settings

All models mentioned in this paper were trained and evaluated on the open-source toolbox PyTorch via the Pycharm platform on a computer with a 12th Gen Intel Core i5-12600KF 3.70GHz CPU and 16GB of installed memory. 3070 GPU (with 8GB of memory) to run all the target detection models involved in the experiments. After a period of experimental exploration, the important hyper parameters in the target detection model are summarized and summarized, and the model achieves stable and reliable performance when the model parameters are set as in Table 5.

Table 5.

Experimental setup

Set item	Parameter
Iteration	300
Batch sizes	16
Initial learning rate	0.1
Min learning rate	0.0001
Optimizer	SGD
Momentum	0.937
Weight decay	5e-4
Learning rate decay type	COS
Thread	4

The set item denotes the type of model hyperparameter; the parameter denotes the value assigned to the experimental hyperparameter in this paper.

Method details

In this section, we first introduce the solder paste printing defect intelligent detection system; then introduce the s-mosica data enhancement algorithm; and finally, introduce the kt-iou loss function. It should be noted that the s-mosica data augmentation algorithm and the kt-iou loss function can be easily generalized to other deep learning models.

Solder paste printing defect detection system

The solder paste printing defect detection system designed in this paper consists of two main structures, and the structure is shown in Figure 10.

Figure 10.

Intelligent detection system for solder paste printing defects

(A) PMP scanning image generation.

(B) defect intelligent detection and display.

As shown in Figure 10, the solder paste printing defect detection system consists of two main parts. The first part is the PMP scanning imaging section shown in (Figure 10A). In this paper, it is confirmed that the industry’s current widely used phase modulation profile measurement technique can measure the complete shape of printed solder paste very accurately and stably, so this technique is retained as a pre-step for the intelligent inspection system of solder in this paper. The second part is (Figure 10B) Deep learning based target detection framework. The functional interface of the upper computer software is shown in Figure 11.

Figure 11.

The functional layout of the upper computer software

The upper computer software uses PyQT5 for interface design and coding, and the underlying logic algorithm is based on the YOLOX³⁰ model and uses the s-mosica data enhancement algorithm and kt-iou loss function proposed in this paper. The functional area of the software is divided into 6 parts as shown in Figure 4, number 1 is the system name, number 2 is the parameter setting area, number 3 is the detection result statistics and display area, number 4 is the original file display area, number 5 is the detection result image display area, and number 6 is the program progress bar. The second part replaces the comparison link in the original SPI system. To check whether there are defects in the object to be inspected in the original SPI system, the usual practice is to compare the reconstructed image of the object to be inspected with the reconstructed image of good quality for visual inspection, and if the comparison is judged to be abnormal, then the technician will confirm the visual inspection, which has a high misjudgment rate and low efficiency. The introduction of a depth target detection model makes the system more intelligent and efficient with a much lower misjudgment rate.

s-Mosica data augmentation algorithm

The mosica algorithm expands the single image training sample into a combination of 4 image samples, which greatly enriches the training sample. Randomly selected 4 images for random scaling, cropping, arranging, and other operations to make the detection dataset rich, especially increasing a lot of small targets, so that the network can learn more small target features, robustness, and generalization performance is better; At the same time, reduce the GPU memory, direct calculation of the 4 image data, so that the Mini-batch size does not need to be very large to achieve better results. In this paper, we believe that although cropping, scaling, and random arrangement increase the number of small targets to enrich the sample, too small targets also bring difficulties in model training and fitting, while increasing the training cost, for example, the target with only a few pixels in length and width has lost information in deeper layers during the convolutional pooling operation of the CNN family of classes, and these very small targets do not end up with computable loss information, but in the The feature extraction process needs to consume a lot of computational resources, and the computational cost increases thus dragging down the speed of model convergence. In this paper, we propose a very small target filtering algorithm to make the enhanced data samples more valuable for training, and the data enhancement algorithm after very small target filtering is named s-mosica. The formula of the very small target filtering algorithm is shown in Equations 1, 2, 3, and 4.

$$W\ge \alpha$$ (Equation 1)

$$H\ge \alpha$$ (Equation 2)

$$\frac{Ao}{An}\ge \beta$$ (Equation 3)

$$\max \left(\frac{W}{H},\frac{H}{W}\right)\le \delta$$ (Equation 4)

where W, and H are the width and height of the enhanced target frame, respectively, α is the pixel threshold; Ao is the area occupied by the enhanced target frame, An is the area occupied by the target frame before enhancement, β is the area threshold occupied by the target frame; δ is the enhanced aspect ratio threshold. After repeated experiments it is determined that α is equal to 3 (pixels), β is equal to 0.1, and δ is equal to 20 with good results. mosica retains most of the targets with training value after threshold brushing, and the comparison before and after brushing is shown in Figure 12.

Figure 12.

Example of transformed image changes between Mosica algorithm to s-mosica algorithm

(A) Mosica Data Enhancement Effect.

(B) S-mosica data enhancement effect.

Observing Figure 12, it is found that (Figure 12A) is the effect of mosica data enhancement, and the yellow arrow in (Figure 12A)) points to a very small target which is retained after random cropping, because the height size of this target is less than 5 pixels, so it is eliminated by the filtering algorithm to become the result in (Figure 12B), it can be observed that the very small target pointed to by the yellow arrow has disappeared, and (Figure 12B) is the optimization result of the s-mosica data enhancement algorithm proposed in this paper.

kt-iou loss function

The target detection task usually consists of classification prediction and regression prediction (prediction of the bounding box).³¹ Early on, L1 or L2 paradigms were used to regress the bbox,³² and later on, iou loss was used as the localization loss.³³ iou loss has the scale-invariant property to the bbox, so it has obvious advantages when training detectors.³⁴ However, the iou loss has a disadvantage in that it suffers from the problem of gradient vanishing when the predicted and real bins do not overlap, which leads to a reduction in convergence speed and detection accuracy. The iou loss variants appeared in the subsequent research, GioU (Generalized IoU) added a penalty term in the IoU loss to mitigate the problem of gradient vanishing,³⁵ DioU (Distance-IoU) and CioU (Complete IoU) took into account the centroid distance and aspect ratio between the predicted frame and the real frame under the penalty condition,³⁶ which solves the problem of gradient disappearance after box overlap and improves the box regression accuracy at the same time, but the deep learning model based on the iou loss function seems to be difficult to learn the features of the essential differences in the samples with high similarity, which are more difficult to differentiate leading to the low regression prediction accuracy, and unable to locate the target efficiently, based on this problem, this paper proposes the kt-iou loss function to solve the problem of difficult training of similar samples, the kt-iou is based on the basic iou loss through a transformation function f to make the original iou change in the interval before and after the threshold t, the penalty term parameter k can adjust the intensity of the change, the main role of the design of the transformation function f is to make the low iou samples get a larger value of the loss in the process of the model training, so as to get the model to pay more attention to the greater training opportunities, kt-iou can be easily generalized to iou and its variants, and has the same effect in models that use iou as the loss function for prediction frame regression. The transformation function f is shown in Equation 5.

$$f=\frac{1}{1+e^{k\left(iou-t\right)}}$$ (Equation 5)

$$loss=1-f\ast iou$$ (Equation 6)

Where, 0 < t < 1, k > 0, the parameter t regulates whether the transform function f acts on the low iou or the high iou, and the parameter k is a penalty term that regulates the strength of the transformation function f. t is the variable of iou. The loss value is adjusted by changing the values of high and low iou, and in this way, the gradient is adjusted to achieve the purpose of differentially training similar features. To observe the loss function relative to the variable iou change rule, the loss function to the variable iou for the derivation of the results is (7) type.

$$\frac{\nabla loss}{\nabla iou}=\frac{\nabla \left(1-f\ast iou\right)}{\nabla iou}=\frac{\nabla \left(1-\frac{1}{1+e^{k\left(iou-t\right)}}\right)}{\nabla iou}=\frac{\left(k\ast iou-1\right)e^{k\ast \left(iou-t\right)}-1}{{\left[1+e^{k\ast \left(iou-t\right)}\right]}^2}$$ (Equation 7)

Let iou = t to obtain Equation 8:

$$\frac{\nabla loss}{\nabla iou}=\frac{1}{4}k\ast t-\frac{1}{2}$$ (Equation 8)

The transformation function f is differentiable everywhere, and when 0<iou<1, the loss function is differentiable everywhere. Because 0 < t < 1 and k > 0, a suitable set of t and k values can always be found, so that the loss function obtains extreme values at t, and monotonically decreases when 0<iou<t, monotonically increases when t<iou<1, and the loss function obtains minimal values when iou = t. The original loss value at t has different monotonicity in the left and right definition domains, and the corresponding gradient will change accordingly. If you want to adjust the loss value when different iou samples are trained, you only need to adjust the value of t in the definition domain of t. To control the intensity of the transformation, you need to adjust the size of k. The larger the value of k, the more drastic the change in the loss value.

Quantification and statistical analysis

Printing solder paste PMP scanning image test set is used to evaluate the performance of the model, the evaluation toolkit to expand a number of specific indicators, which contains the average correct rate (AP) to assess the target detection of each class is good or bad; mAP is the average of the correct rate of all the classes; Recall (Recall) measures the ability of the model to find all the correct targets, the larger the value of the model the better the performance; F1-score evaluation of the The F1-score evaluates the reconciled average of mAP and Recall, in this paper, the threshold is set to 0.5, the larger the F1-score is the higher quality of the model.All statistical analyses were conducted using Python (version 3.8.8 (Scikit-learn library, version 0.21.1; Matplotlib library, version 3.7.4; numpy, version 1.23.1).

Published: February 5, 2024