Multi-representation thermal features for enhanced defect analysis in pulse thermography

Abstract

AI-driven pulse thermography (PT) has become a crucial tool in non-destructive testing (NDT), enabling automatic detection of hidden anomalies in various industrial components. Current state-of-the-art PT segmentation and depth estimation networks rely on thermal features extracted from Principal Component Thermography (PCT) or Thermographic Signal Reconstruction (TSR) representations. However, relying on PCT and TSR independently constrains the performance of PT inspection models, as these representations possess complementary semantic features. To address this limitation, this work proposes PT-Fusion, a novel feature fusion network for enhanced analysis of subsurface defects in PT setups. PT-Fusion introduces novel fusion modules, Adaptive Weighing Fusion Gate (AWFG) and Gating Enhanced Decoding Block (GEDB), to adaptively fuse thermal features extracted from PCT and TSR representations. A novel data augmentation technique is also proposed based on random data sampling from thermographic sequences to address the scarcity of PT datasets. PT-Fusion is benchmarked against state-of-the-art PT inspection models, including U-Net, attention U-Net, 3D-CNN, TransUNet, and Swin-UNet on the Université Laval IRT-PVC dataset. The results demonstrate that PT-Fusion outperforms U-Net, attention U-Net, and 3D-CNN architectures in defect segmentation and depth estimation accuracies with a margin of 10%. Compared to TransUNet and Swin-UNet, the results show that PT-Fusion’s performance is on par with the aforementioned models, with fewer parameters. Video Link.

Introduction

Over recent years, the growing demand for quality control has driven the emergence of advanced inspection processes in various production lines, such as the aerospace^1,2, automotive^3,4, and construction⁵ industries. This paradigm shift led to the adoption of Non-Destructive Testing (NDT) techniques to detect subsurface defects and assess the structural integrity of industrial components. As a result, a wide range of NDT techniques have been developed to detect subsurface defects, including radiography, ultrasonic testing, and infrared thermography (IRT). These methods provide reliable evaluation of materials and assemblies without impairing their functionality, ensuring the safety and longevity of critical components. Moreover, the continuous advancement of sensing technologies and data-driven algorithms has enabled higher sensitivity and automation in NDT inspection processes, making them indispensable in modern manufacturing environments.

IRT has garnered significant attention due to its simplicity, low cost, and non-contact nature⁶. These advantages have demonstrated the potential of IRT for inspection of aircraft parts^7–10, construction pipelines^5,11–13, and artworks^14,15. Among IRT instrumentation setups, pulse thermography (PT) is valued for its efficiency in inspecting large areas and its capability to rapidly identify subsurface defects¹⁶. Meanwhile, recent advancements in artificial intelligence (AI) have been adopted in PT setups to enhance the quality of the extracted features to improve inspection accuracies. For instance, AI-driven PT inspection algorithms have been proposed for defect classification^17,18, segmentation^19,20, and depth estimation^2,9,21. In addition, the integration of deep learning with thermal signal processing has enabled more robust interpretation of complex thermal transients, reducing dependency on manual thresholding or handcrafted features^19,22–24. Such integration not only enhances the detection sensitivity but also contributes to developing fully automated thermographic inspection frameworks suitable for large-scale industrial applications.

State-of-the-art AI-based PT methods rely on features extracted from principal component thermography (PCT)²⁵ and thermal signal reconstruction (TSR)²⁶ representations for defect analysis. While these representation paradigms demonstrated considerable potential, extracting features from PCT and TSR independently limits the performance of PT inspection models. For instance, PCT tends to enhance defect visibility while TSR models the temporal evolution of the pixel responses. To address this limitation, this work introduces PT-Fusion, a novel fusion network that fuses features extracted from PCT and TSR complementary representations for enhanced subsurface defect analysis in PT setups. The contributions of the paper are the following:

1. PT-Fusion network architecture is proposed for defect segmentation and depth estimation in pulse thermography setups.

2. Novel feature fusion modules, Adaptive Weighing Fusion Gate (AWFG) and Gating Enhanced Decoding Block (GEDB), are proposed to fuse thermal features extracted from PCT and TSR representations for enhanced segmentation and depth estimation of subsurface defects.

3. A novel data augmentation technique is introduced, leveraging random sampling from thermographic sequences to overcome the burden of complex PT data collection setups.

4. The proposed work is tested on the IRT-PVC dataset and benchmarked against state-of-the-art PT inspection models. The results demonstrate that PT-Fusion outperforms U-Net, attention U-Net, and 3D-CNN, while on par with transformer-based architectures in segmentation and depth estimation accuracies.

Related work

PT has established itself as a key NDT technique for the detection of subsurface defects in industrial components. In addition, the recent advancements in artificial intelligence (AI) have significantly enhanced PT methodologies, enabling automated defect analysis²⁷. As a result, extensive investigations have emerged on AI-driven PT leveraging learning-based features, with various architectures proposed for defect detection, segmentation, and depth estimation. For instance, a broad spectrum of defect detection models has been developed, demonstrating the potential of PT. Muller et al.²⁸ introduced a ConvLSTM network for the detection of flat defects in forged steel parts. Consequently, Ankel et al.²⁹ extended convolutional neural networks (CNNs) for elliptical defect detection. Other state-of-the-art detection networks³⁰ have been developed for defect detection, such as YOLOv5²¹ and Faster R-CNN¹⁷. Since these methods are purely convolutional, the quality of features extracted by CNN-based architectures is limited by the network kernels’ receptive fields. Thus, an attention-based defect detection network was introduced to simultaneously learn local and global features, circumventing the limited receptive fields of CNNs²³. Other methods leverage generative models for data augmentation and progressive training for improved learning of defect detection networks^31,32.

Similarly, a wide stream of defect segmentation and depth estimation networks have been proposed relying on learning-based features in the context of PT. Fang et al.³³ performed an experimental study on state-of-the-art network architectures, such as U-Net and ResNet, for subsurface defect segmentation. Zhou et al.¹⁹ introduced the attention U-Net for enhanced defect segmentation compared to its traditional U-Net counterpart. On the other hand, Morelli et al.³⁴ developed a residual convolutional network for heat-invariant defect segmentation in lock-in thermography. The aforementioned networks are inherently 2D with minimal consideration of the significant temporal information embedded in thermographic sequences. Thus, Dong et al.² proposed a 3D-CNN accounting for temporal information embedded in raw thermal sequences for enhancing defect segmentation accuracies. However, all these networks are completely convolutional, and the limited receptive field of CNNs degrades the quality of the constructed feature maps. Consequently, several works leveraged attention-based segmentation architectures to extract features rich with global spatial and temporal features. Schmid et al.³⁵ and Xie et al.³⁶ proposed temporal attention mechanisms for improving learned features from temporal sequences. In addition, an attention-based PHM-IRNET network architecture was proposed as a self-supervised learning framework for segmentation in industrial thermographic inspection³⁷.

In all of the aforementioned learning-based frameworks, networks learn to extract thermal features from single representations such as raw thermal sequences, pulse-phase thermography (PPT)^38,39, PCT^25,40,41, or TSR^26,42,43. Learning-based representations also exist^14,22, but also act as single inputs to deep neural networks. However, the aforementioned representations possess semantic features that are rather complementary. For instance, PCT tends to enhance defect visibility while TSR models the temporal evolution of the pixel responses. Unlike state-of-the-art methods, this work proposes learning these complementary features in an end-to-end fusion network, PT-Fusion. PT-Fusion passes PCT and TSR representations into CNN heads, and the extracted feature maps are fed to novel feature fusion blocks, AWFG and GEDB, to dynamically learn spatial and temporal features. The fused features serve as our multi-representation thermal features and are decoded for enhanced defect segmentation and depth estimation.

The rest of the article is structured as follows. Results section presents the experimental validations of the proposed methodology tested on the IRT-PVC dataset. The Methods section outlines the proposed fusion network and data augmentation strategy. Finally, Conclusions section presents conclusions, findings, and future aspects of the proposed framework.

Results

IRT-PVC dataset

The work of this paper is evaluated on the IRT-PVC dataset²¹. The IRT-PVC dataset is a publicly available pulse thermography dataset designed for subsurface defect analysis and depth estimation on PVC samples. The dataset comprises 38 inspection sequences collected from 3D-printed PVC specimens containing back-drilled artificial defects at controlled depths. The defects are manufactured in two geometries, circles and squares, and at depths between 2.5 mm and 4.5 mm. Each inspection sequence consists of a pulsed heating followed by passive cooling, captured using a FLIR SC5000 infrared camera at a spatial resolution of 320256 pixels with a 10 Hz acquisition frequency for a duration of 181 s, producing approximately 1810 frames per sample. Heating is applied using two 4 kJ xenon flash lamps, synchronized through a computer-controlled trigger to ensure reproducible thermal excitation conditions. For the groundtruth segmentation masks, each specimen in the dataset is accompanied by precise defect location and depth labels that were generated using a refined 3D–CAD-assisted labeling procedure. The CAD-based method provides pixel-accurate defect masks by projecting the 3D geometry of each specimen onto the infrared image plane and aligning it with the corresponding PCT frames. This alignment enables direct extraction of the true defect size and depth, resulting in high labeling accuracy for reliable evaluation.

Experimental validation

The proposed PT-Fusion network is validated on the IRT-PVC inspection sequences comprising 38 inspection sequences of PVC specimens with back-drilled holes at different depths ranging between 2.5 and 4.5 mm. 26 sequences are for training, 6 for validation, and 6 for testing. The performance metrics of PT-Fusion are reported on the validation and testing samples, as reported in²¹. We test two variants of the network on the original IRT-PVC inspection sequences and the proposed augmentation technique is only utilized during training. The first variant is trained to segment defects to different depth classes; thereby, multi-class segmentation. For such a type of problem, the common evaluation metrics, mean Intersection over Union mIoU, Recall, and Precision, are employed for evaluation and are defined by

1

2

3

where and are the predicted and ground truth sets of pixels for class c, respectively. Similarly, TP, FP, and FN are the predicted true positives, false positives, and false negatives, respectively. Note that the IRT-PVC dataset samples comprise defects at depths of 2.5, 3.0, 3.5, 4.0, and 4.5 mm. Accordingly, each defect depth is treated as a class and multi-class classification is carried out to classify each pixel belonging to either of the defect classes or sound background.

The second variant of the network is trained for simultaneous binary segmentation and depth estimation. Thus, the evaluation metrics are the segmentation IoU, mean absolute error (MAE), root mean square error (RMSE), and correlation coefficient, r, for depth, defined as

4

5

6

7

where P and G are the predicted and ground truth segmentation masks, and are the ground truth and predicted depth values, respectively, N is the total number of pixels, and and denote the mean ground truth and predicted depths. It is worth mentioning that the two testing routines also present benchmarks of PT-Fusion against state-of-the-art defect segmentation and depth estimation models. In addition, all the baseline networks are trained on the augmented IRT-PVC dataset using the proposed augmentation strategy and tested on the IRT-PVC dataset testing sequences for fair comparisons. Finally, the effect of the proposed augmentation technique is discussed and a generalizibility study is presented to evaluate the generalization performance of PT-Fusion.

Defect segmentation evaluation

The straightforward approach for defect segmentation using multiple representations is by concatenating the PCT and TSR images at the input. However, we show that such a black box implementation does not provide optimal learning performance for the network. In general, it is always beneficial to guide the network architecture to inherently learn the contributions of the representations and dynamically fuse the extracted thermal feature maps. Table 1 shows PT-Fusion’s performance with respect to the defect classes. Note that the IRT-PVC samples contain defects at different depths of 2.5, 3.0, 3.5, 4.0, and 4.5 mm and each are treated as a separate defect type when reporting the segmentation metrics, mIoU, recall, and precision. On the other hand, Table 2 demonstrates qualitative comparisons of the predicted segmentation masks by a black box U-Net²⁰, attention U-Net¹⁹, 3D-CNN², Swin-UNet⁴⁴, TransUNet⁴⁵, and PT-Fusion. The visualized results are also quantified in Table 3 in terms of the mIoU, recall, and precision, where PT-Fusion is also benchmarked against state-of-the-art defect segmentation networks. Note that all reference methods are deep architectures and require large number of training samples. As such, all baseline networks have been trained on the augmented dataset for stable training and fair comparisons.

Table 1.

Performance of PT-Fusion for multi-class segmentation on the IRT-PVC dataset, where each defect depth is treated as a defect class.

Metric	Background	2.5 mm	3.0 mm	3.5 mm	4.0 mm	4.5 mm	All
mIoU	0.889	0.852	0.875	0.879	0.863	0.856	0.869
Precision	0.924	0.879	0.911	0.920	0.911	0.898	0.907
Recall	0.941	0.919	0.946	0.957	0.942	0.930	0.939

Table 2.

Qualitative comparisons between state-of-the-art networks for multi-class defect segmentation, U-Net, attention U-Net, 3D-CNN, Swin-UNet, and TransUNet against PT-Fusion.

Method	Input	Sample #1	Sample #2	Sample #3
U-Net	PCT-TSR Concatenated
Attention U-Net	PCT-TSR Concatenated
3D-CNN	Sequence
Swin-UNet	PCT-TSR Concatenated
TransUNet	PCT-TSR Concatenated
Ours	PCT-TSR Fused
GT	N/A

Table 3.

PT-Fusion benchmarked against state-of-the-art defect segmentation architectures in terms of mIoU, Recall, and Precision, including Attention U-Net¹⁹, 3D-CNN², Swin-UNet⁴⁴, and TransUNet⁴⁵.

Method	Input	mIoU	Recall	Precision
Attention U-Net	PCT	0.721	0.696	0.722
Attention U-Net	TSR	0.711	0.708	0.723
Attention U-Net	PCT–TSR (Concat.)	0.804	0.827	0.864
U-Net	PCT–TSR (Concat.)	0.766	0.801	0.813
3D-CNN	Sequence	0.784	0.821	0.847
Swin-UNet	PCT–TSR (Concat.)	0.867	0.861	0.897
TransUNet	PCT–TSR (Concat.)	0.873	0.884	0.912
Ours	Fused PCT–TSR	0.869	0.907	0.939

The obtained results outline several key findings. First, PT-Fusion demonstrates consistent results across all defect depths as highlighted in Table 1. This confirms that the model maintains robustness in predicting both shallow and deeper subsurface defects, avoiding bias toward a specific depth range. Second, according to Table 3, model performances fall around 0.7 when trained on either PCT or TSR representations. When these representations are concatenated at the input, the mIoU increases to 0.766 for U-Net and 0.804 for attention U-Net. In addition, a increase in the recall and precision is witnessed. However, concatenation at the input does not ensure optimal model performance. Hence, PT-Fusion architecture incorporating the fusion modules, AWFG and GEDB, effectively guides the network learning performance towards enhanced feature extraction and defect segmentation, where it outperforms the aforementioned methods by achieving an mIoU of 0.869, a Recall of 0.907, and a Precision of 0.939. Note that the attention U-Net with a single input resembles a PT-Fusion network without multi-representation inputs as no fusion blocks are required. As such, as a nearly increase is witnessed across all metrics compared to the attention U-Net, this demonstrates that the features from both representations are indeed complementary. For instance, the proposed architecture inherently learns to adaptively identify the weighted contribution of the extracted features from PCT and TSR representations to achieve state-of-the-art segmentation accuracy. Other methods, such as 3D-CNN, leverage 3D convolutions to extract spatial and temporal information simultaneously. Still, the network performance falls short compared to PT-Fusion. We hypothesize that this is due to the engineered sequence compression of the input discussed in². This highlights that PT-Fusion learns to leverage the enhanced defect visibility from PCT images and the temporal information of the pixel responses modeled in TSR in an efficient and enhanced multi-representation thermal features. Moreover, compared to transformer-based architectures, Swin-UNet and TransUNet , PT-Fusion outperforms Swin-UNet across all metrics with fewer parameters. For instance, Swin-UNet possesses nearly 69.81M parameters while PT-Fusion possesses 24.09M parameters. Compared to TransUNet, PT-Fusion falls slightly short in terms of mIoU but outperforms the aforementioned network in terms of recall and precision with approximately quarter the number of paramaters, i.e. TransUNet comprises of 109.54M parameters. Table 4 compares the computational demands of Swin-UNet and TransUNet against PT-Fusion in terms of number of learnable parameters, FLOPs, and inference time on RTX 4070 GPU for input size of . The metrics highlight that our framework is capable of harnessing robust thermal features efficiently while maintaining the required segmentation accuracy. Table 5 also benchmarks the total inference time of PT-Fusion, including the computational overhead of computing the PCT and TSR representations using CPU and GPU. Note that in the PT-Fusion implementation, the PCT and TSR representations are computed on the GPU to reduce computational overhead, since generating these representations on large thermography sequences requires higher processing time compared to the network’s inference time.

Table 4.

Computational demands of Swin-UNet and TransUNet compared to PT-Fusion.

Metric	Swin-UNet	TransUNet	Ours
Parameters (M)	69.81	109.54	24.09
FLOPs (G)	27.43	85.96	15.24
Inference Time (ms)	28.54	44.05	10.91

Table 5.

Total execution time of PT-Fusion benchmarked against Swin-UNet and TransUNet.

Computational Block	PCT	TSR	PT-Fusion	Swin-UNet	TransUNet
Intel i7–12700K CPU (s)	7.37	8.21	0.0482	0.128	0.337
RTX 4070 GPU (s)	2.19	2.46	0.0109	0.0785	0.141
Total CPU (s)	–	–	15.62	15.71	15.92
Total GPU (s)	–	–	4.66	4.73	4.79

Depth estimation performance

We also evaluate the PT-Fusion architecture for depth estimation. While the previous evaluation routine involved multi-class segmentation, simultaneous binary segmentation and depth prediction are more intuitive in actual inspection scenarios. This motivated training the second variant of the proposed network that classifies the input pixels as either defective or sound, along with the defect depth. Figure 1 shows the predicted binary segmentation masks and depth maps for three samples in the IRT-PVC dataset. Table 6 presents the segmentation and depth estimation of PT-Fusion based on the IoU, MAE, RMSE, and correlation coefficient. In addition, Table 7 benchmarks PT-Fusion’s performance against U-Net, attention U-Net, 3D-CNN, Swin-UNet, and TransUNet architectures in a similar fashion to the defect segmentation evaluation. All baseline methods are trained on the augmented dataset for fair comparisons similar to the defect segmentation evaluation benchmarks.

Fig. 1.

Simultaneous segmentation and depth estimation predictions by PT-Fusion for 5 samples (a–e). The segmentation and depth predictions, illustrated in the first and second rows, respectively, are compared against the ground truth in the third row.

Table 6.

Obtained defect segmentation and depth estimation performance of PT-Fusion on each defect class from the IRT-PVC dataset.

Defect Class	mIoU	MAE (mm)	RMSE (mm)	Corr. Coeff.
2.5 mm	0.883	0.0068	0.0096	0.990
3.0 mm	0.888	0.0091	0.0109	0.986
3.5 mm	0.893	0.0104	0.0108	0.989
4.0 mm	0.877	0.0076	0.0106	0.989
4.5 mm	0.868	0.0101	0.0114	0.985
All	0.882	0.0082	0.0107	0.988

Table 7.

PT-Fusion benchmarked against state-of-the-art architectures for simultaneous defect segmentation and depth estimation in terms of IoU, MAE, RMSE, and correlation coefficient.

Method	U-Net	Att. U-Net	Att. U-Net	Att. U-Net	3D-CNN	Swin-UNet	TransUNet	Ours
Input	Concat.	PCT	TSR	Concat.	Sequence	Concat.	Concat.	Fused
IoU	0.828	0.748	0.787	0.839	0.823	0.847	0.871	0.882
MAE (mm)	0.0108	0.0120	0.0119	0.0103	0.0113	0.0103	0.0078	0.0082
RMSE (mm)	0.0289	0.0331	0.0294	0.0149	0.0202	0.0136	0.0119	0.0107
Corr. Coeff.	0.932	0.889	0.911	0.963	0.947	0.977	0.981	0.988

Similar findings to the defect segmentation evaluation can be drawn. The IoU for all networks shows an improvement compared to the mIoU in Tables 3 and 6. This is expected since binary segmentation tends to be a simplified version of multi-class segmentation. In addition, since the network predicts the segmentation mask and depth map simultaneously, the segmentation performance is improved as it is guided by the predicted depth of the defect. The obtained MAE and RMSE for PT-Fusion show that the network achieves robust performance with both low average error and limited variance across the dataset. The close values of MAE and RMSE indicate that predictions are consistently accurate without significant outliers. Additionally, PT-Fusion’s high correlation coefficient shows that predicted depths closely follow the ground truth demonstrating that the model captures the physics of the underlying heat diffusion process in the inspected sequences. On the other hand, by comparing the individual performances of the networks, PT-Fusion outperforms the state-of-the-art network architectures by adaptively computing the contributions of multi-representation thermal features. This is also evident in the depth estimation task since it heavily relies on the pixels’ temporal responses along with the spatial features in the representations. Even for strong transformer-based architectures such as Swin-UNet, PT-Fusion shows stronger segmentation and depth estimation accuracy across all metrics, while on TransUNet, PT-Fusion shows lower RMSE and higher correlation coefficient with fewer number of learnable parameters. Hence, PT-Fusion learns to extract multi-representation thermal features, harnessing the spatial and temporal information embedded in TSR and PCT representations, leading to improved segmentation and depth estimation.

Effect of data augmentation

The obtained network performance is heavily dependent on the diversity of the dataset. We demonstrate the need for the proposed augmentation strategy by training PT-Fusion on the original IRT-PVC dataset augmented using the common augmentation techniques, i.e., random translations, rotations, and shearing. Figure 2a shows the training loss for PT-Fusion trained on the original dataset. As depicted in the figure, the validation loss tends to remain constant and does not decrease with the training loss. The training resulted in a corresponding testing IoU of 0.567. This implies that the traditional augmentation techniques are still not sufficient to train the deep architecture of PT-Fusion. In contrast, Fig. 2b shows the training loss for PT-Fusion trained on the IRT-PVC sequences augmented using the introduced augmentation approach along with the aforementioned common augmentation techniques. Table 8 also quantifies PT-Fusion’s performance when trained on the IRT-PVC dataset augmented and unaugmented datasets for simultaneous binary segmentation and depth estimation using the proposed augmentation strategy. Note that when training on the unaugmented dataset, spatial augmentation methods are applied during training, i.e. random cropping, rotations, etc. The devised data augmentation strategy proves to be sufficient to grant PT-Fusion the required generalization and segmentation performance, where the obtained testing IoU is 0.882. In contrast, PT-Fusion’s achieves an IoU of 0.567 and MAE of 0.128 mm when trained on the unaugmented dataset. While traditional augmentation methods tend to virtually increase the sizes of the datasets, the augmentation is done spatially. In addition, PT-based inspection data analysis inherently involves spatiotemporal data requiring spatial and temporal augmentations simultaneously. Hence, our data augmentation method incorporates spatial and temporal augmentations, ensuring dataset diversity and improved training of the PT-Fusion network.

Fig. 2.

Training loss for PT-Fusion trained on a data augmented with common augmentation techniques and b data augmented using the proposed augmentation strategy.

Table 8.

PT-Fusion’s performance when trained on the augmented and unaugmented dataset, highlighting the effect of the proposed data augmentation technique.

Dataset Type	mIoU	MAE (mm)	RMSE (mm)	Corr. Coeff.
Augmented	0.882	0.0082	0.0107	0.988
Unaugmented	0.567	0.128	0.347	0.659

Ablation studies

We perform four further studies on the network architecture trained for simultaneous segmentation and depth estimation: 1) The weighing factor in the loss function , 2) incorporating self-attention mechanisms instead of the proposed feature fusion modules, AWFG and GEDB, for harnessing long-range dependencies in the encoded features, 3) studying the effect of replacing PCT or TSR with pulse-phase thermography representation, and 4) studying the effect when changing the number of PCT and TSR in the input representations. The studies are based on the IoU metric for binary segmentation and MAE for depth estimation. With regards to , the MAE tends to decrease with increasing lambda till . Then, the MAE remains approximately consistent. In contrast, the IoU tends to remain approximately 0.88 till and starts to decrease dramatically. This is because the loss function importance shifts to the depth estimation task. Hence, based on Fig. 3, one can infer that the optimal value of is 0.5, balancing the accuracy trade-off between the MAE and IoU.

Fig. 3.

Effect of the weighing factor on IoU and MAE, where provides the optimal trade-off between both metrics and is utilized during training.

The second study involves incorporating self-attention and cross-attention mechanisms in combination with or in place of AWFG and GEDB feature fusion blocks. Self-attention and cross-attention mechanisms allow the model to capture long-range dependencies between the obtained feature maps from PT-Fusion encoders. Table 9 compares PT-Fusion performance with and without the attention mechanisms in terms of IoU and MAE. First, it can be inferred that when combining attention mechanisms with GEDB and AWFG, no significant improvement in performance is witnessed in terms of IoU and depth MAE. On the other hand, by fully employing the attention mechanisms as fusion modules, improvements in the obtained performances are approximately with a 5-fold increase in the number of trainable parameters. This implies that learned features required for inspection of the IRT-PVC samples are mostly local. The reason behind this is that the PT setup involves the inspection of PVC samples, which are generally heat insulators. Hence, heat remains locally confined within the defects, with no global features in the input thermographic representations. This highlights that AWFG and GEDB feature fusion modules are sufficient to capture these features for defect segmentation and depth estimation.

Table 9.

Ablation study comparing fusion configurations for PT-Fusion in terms of IoU and MAE.

Fusion Modules	IoU	MAE (mm)
AWFG + GEDB	0.882	0.0082
Self-Attn + Cross-Attention	0.901	0.0072
AWFG + Cross-Attention	0.883	0.0080
Self-Attention + GEDB	0.884	0.0084
AWFG + GEDB + Self-Attention	0.888	0.0079

The third study revolves around incorporating PPT representation in place of PCT and TSR inputs, or along these representations. PPT is a common thermography representation that performs a frequency-domain analysis of the transient thermographic sequence, where the phase component is known to be relatively invariant to non-uniform heating and surface emissivity variations. Table 10 outlines the performance of PT-Fusion when PPT is utilized as input in place of PCT and TSR or in addition to the aforementioned representations. The obtained results show that when combining PPT with PCT, the performance remains on the same order of magnitude as when using TSR with PCT. This is expected, since PPT models the heat-diffusion dynamics in PT in the frequency domain. In contrast, fusing PPT with TSR does not lead to similar improvements; the obtained metrics are close to those of the attention U-Net when using TSR alone. This is because TSR and PPT capture similar thermophysical information due to the duality between temporal and frequency domain representations of the heat-diffusion process. Finally, when combining PPT, TSR, and PCT in PT-Fusion, no significant improvements are obtained and the results remain consistent compared to combining PCT and TSR only. In other words, the addition of PPT on top of PCT and TSR can be interpreted as a saturation of input representations, where the PCT–TSR pair already provides the optimal trade-off between efficiency and accuracy for PT inspection tasks.

Table 10.

PT-Fusion performance when incorporating PPT representation in place of or alongside PCT and TSR.

Input Representation	mIoU	MAE (mm)	RMSE (mm)	Corr. Coeff.
PCT + TSR	0.882	0.0082	0.0107	0.988
PCT + PPT	0.873	0.0084	0.0111	0.987
TSR + PPT	0.802	0.0116	0.0173	0.965
PCT + TSR + PPT	0.887	0.0080	0.0102	0.988

The final study involves testing the sensitivity of PT-Fusion to the choice of representation hyperparameters when running simultaneous defect segmentation and depth estimation inspection. Meaning, we evaluate the effect of varying the number of principal component thermography (PCT) components used as input, ranging from the first 5 components up to 10. In addition, we assess the network’s sensitivity to the TSR representation by comparing logarithmic polynomial fitting of degree 4 and degree 5. Table 11 shows the obtained defect segmentation and depth estimation performance of PT-Fusion when utilizing PCT components ranging from 5 to 10, while relying on a TSR of degree 5. The obtained results show that the performance of the network tends to be consistent across different number of PCT components, with IoU improving marginally from 0.876 at 5 components to 0.882 at 10 components, while depth estimation errors gradually decrease with increasing PCT components, as MAE decreases from 0.0099 mm at 5 components to 0.0082 mm at 10 components, and RMSE reducing from 0.0122 mm to 0.0107 mm. The marginal improvement in performances shows that PT-Fusion tends to leverage defect-relevant information when increasing the number of PCT components, even when introducing noisy principal components, 6-10. With regards to the TSR representation polynomial degree hyperparameter, Table 12 outlines the obtained depth estimation performances across different TSR fitting degrees and 10 PCT components. A degree 5 logarithmic fitting provides a slight improvement over degree 4. This highlights that a polynomial degree of 5 tends to better model the heat diffusion physics of the inspection sequence, while polynomial fitting of degree 4 tends to slightly underfit. Nevertheless, PT-Fusion tends to provide segmentation and depth estimation accuracies at the same order of magnitude in both cases.

Table 11.

PT-Fusion performance when using different numbers of PCT components, showing that the network remains robust even when additional noisy components are introduced.

No. PCT Components	5	6	7	8	9	10
IoU	0.876	0.877	0.880	0.881	0.881	0.882
MAE (mm)	0.0099	0.0095	0.0083	0.0083	0.0082	0.0082
RMSE (mm)	0.0122	0.0121	0.0114	0.0109	0.0108	0.0107
Corr. Coeff.	0.972	0.972	0.979	0.984	0.988	0.988

Table 12.

Ablation study on TSR logarithmic polynomial fitting degree showing consistent PT-Fusion performance.

TSR Degree	4	5
IoU	0.879	0.882
MAE (mm)	0.0083	0.0082
RMSE (mm)	0.0111	0.0107
Corr. Coeff.	0.987	0.988

Generalizibility evaluation

In addition to the conducted ablation studies, we further evaluate the generalization capability of PT-Fusion with respect to defect geometries. The IRT-PVC dataset comprises 38 samples, where 18 have square shape defects and 18 have circular defects. Accordingly, PT-Fusion was trained using the augmented dataset of one defect type, tested on the other, and vice versa, simulating a scenario where defect geometries encountered during testing are not seen during training. This study was essentially added to test whether our framework tends to overfit to defect shapes or can generalize to defects of unseen geometries during training. Table 13 summarizes the obtained binary segmentation and depth estimation performances. When trained on square defects and tested on samples with circular defects, PT-Fusion achieved an IoU of 0.857, MAE of 0.0096 mm, RMSE of 0.0112 mm, and correlation coefficient of 0.985. Similarly, training on circular defects and testing on samples of square defects resulted in an IoU of 0.853, MAE of 0.0089 mm, RMSE of 0.0119 mm, and correlation coefficient of 0.983. In both testing scenarios, the proposed framework demonstrates the ability to generalize without overfitting to defect geometric priors during and training and preserves consistent performance on unseen defect geometries during testing. In addition, the network shows consistent orders of magnitude across all metrics when trained on either defect type.

Table 13.

Generalizability evaluation of PT-Fusion on the IRT-PVC dataset, demonstrating that the proposed framework does not overfit to defect geometries.

Training Dataset	Square	Circular
Testing Dataset	Circular	Square
IoU	0.857	0.853
MAE (mm)	0.0096	0.0089
RMSE (mm)	0.0112	0.0119
Corr. Coeff.	0.985	0.983

Discussion

The findings presented in this paper highlight the effectiveness of PT-Fusion in leveraging complementary thermal representations for accurate and reliable defect analysis. By jointly learning from PCT and TSR inputs, the network successfully integrates spatial contrast enhancement with temporal heat-diffusion dynamics, resulting in more discriminative feature representations. This is highlighted in both segmentation and depth-estimation tasks, where PT-Fusion consistently outperforms conventional architectures such as 3D-CNN and attention U-Net. On the other hand, compared to transformer-based baselines, PT-Fusion is able to perform on part with SwinUNet and TransUNet with significantly fewer computational requirements. In addition to the obtained defect analysis accuracies, a key strength of the proposed framework lies in its ability to generalize beyond the specific defect geometries seen during training. The cross-geometry experiments demonstrate that PT-Fusion captures the underlying thermophysical behavior of heat propagation rather than memorizing superficial geometric cues, a property critical for real-world inspection scenarios. Furthermore, the results confirm that the introduced spatiotemporal augmentation strategy enhances data diversity, mitigates overfitting, and improves robustness under limited dataset conditions, a critical challenge in NDT research. Finally, the ablation studies reveal that the AWFG and GEDB fusion modules balance local feature extraction and long-range dependency modeling without the excessive computational burden. This design ensures scalability and practical deployability in industrial settings where inference speed is crucial.

Pulse thermography

A typical PT setup is shown in Fig. 4. PT involves a flash lamp providing a heat pulse into a target specimen⁴⁶. If the specimen is sound, all pixels approximately generate the same thermal profile. In the presence of defects, heat gets trapped, generating an abnormal thermal distribution that is captured over time by the thermal camera. Consequently, the acquired thermal images are pre-processed using the methods discussed in the following section and are utilized as inputs to PT-Fusion. Several public datasets are available following the general PT setup discussed in Fig. 4. In this work, PT-Fusion is trained and tested on the IRT-PVC dataset²¹. The dataset comprises 38 inspection sequences of 3D-printed PVC specimens with back-drilled holes at different depths ranging between 2.5 and 4.5 mm. Each inspection sequence is a 3D matrix of shape with thermograms, where is the image height, is its width, is the image timestamp, and i represents the sample index. is reshaped to by a raster-like operation and standardized by

8

where,

9

10

and is the standardized pixel-wise thermal response. Developing PT inspection algorithms that process the entire thermal sequence introduces significant computational complexities. Instead, thermal features are extracted from compressed PT sequences such as PCT and TSR representations. These representations offer the advantage of reducing data dimensionality while preserving the most important features for defect detection. PCT images enhance defect clarity, while TSR images incorporate the temporal evolution of the pixel responses.

Fig. 4.

PT setup involving flash lamps providing heat pulse to a specimen. Defective areas trap heat flow generating abnormal temperature distributions.

To generate PCT images, singular value decomposition (SVD) is applied by

11

where comprises empirical orthogonal functions representing spatial differences within the input pixel responses, is a diagonal matrix comprising singular eigenvalues in descending order, while matrix V contains their corresponding principal components. Consequently, the principal component images are computed by

12

where is the eigenvector in the matrix V. It is worth mentioning that only the first 5-10 principal component images are utilized for inspection²⁵, where in our study is selected to generate the PCT image tensor , and .

On the other hand, TSR image generation involves logarithmic polynomial fitting on the pixel’s thermal response with an n-degree polynomial as

13

where the coefficients of the n-degree polynomial, is the temperature increase, and t is time. 4-5 degree polynomials are sufficient to model the pixel responses. In this work, an degree polynomial is employed. The resulting polynomial coefficients are used as the values for the TSR image channels, forming the TSR image tensor and . Collectively, each sequence generates a 10 channel PCT image and 5 channel TSR image, and are utilized as inputs to the proposed PT-Fusion network.

Methods

Overview

The direct approach for fusing thermal features from both representations is by either concatenating them at the input or concatenating their feature maps in the intermediate layers. However, the network, in such cases, does not inherently learn the weighted contribution of the extracted features; thereby, limiting its performance. PT-Fusion, shown in Fig. 5, learns to adaptively integrate the complementary features in PCT and TSR images. Given a pulse thermography sequence. PT-Fusion processes these representations using two parallel CNN encoder branches to extract features from each input representation. The encoded feature maps are then integrated using two fusion modules: the Adaptive Weighing Fusion Gate (AWFG), which combines PCT and TSR features at the bottleneck, and the Gating Enhanced Decoding Block (GEDB), which fuses the multi-representation features with the decoder outputs throughout the decoding path. The fused features are finally decoded to generate pixel-wise predictions for either multi-class defect segmentation or simultaneous binary segmentation and defect depth estimation. To train the deep architecture of PT-Fusion and alleviate the scarcity of thermography datasets, a spatiotemporal augmentation strategy is introduced to diversify the thermal responses. Given a raw PT sequence, temporal subsampling is first applied to simulate variations in heat diffusion rates and acquisition durations. Additive noise perturbations are then injected to mimic sensor noise and acquisition artifacts encountered in real inspection settings. The proposed augmentation technique preserves the underlying thermal diffusion characteristics of the defect while generating a wider distribution of thermal appearances, enabling PT-Fusion to learn robust spatial–temporal thermal features under limited data availability.

Fig. 5.

a PT-Fusion network architecture. PCT and TSR images are fed to a shallow CNN head. b The fusion module, Adaptive Weighing Fusion Gate (AWFG), is placed at the output of the encoders, and c the Gating Enhanced Decoding Block (GEDB) is added to adaptively fuse the encoded, multi-representation features with the decoders output.

PT-fusion architecture

PT-Fusion, illustrated in Fig. 5, follows a U-Net encoder-decoder architecture⁴⁷ and involves passing the PCT image representation, , and TSR image representation, , through two parallel CNN heads to initially extract feature maps from both representations. Each CNN head consists of m encoder layers, where . Each layer, see Fig. 6, feeds its input through a learnable residual convolution with a downsampling max pooling operation as

14

15

where and are the m-th encoder layer outputs of the PCT and TSR CNN heads. Similarly, and are the layer inputs received from the encoder layer.

Fig. 6.

Encoder layer m of the PT-Fusion CNN heads involves a residual convolution block and downsampling max pooling for extracting feature maps.

After encoding the input from the PCT and TSR representations to feature maps and , the features are first fused in AWFG, outlined in in Algorithm 1, using their weighted combination to exploit the complementary features. AWFG is governed by learnable adaptive weights , derived from the PCT features as

16

where represents a convolution, while is the sigmoid activation. The sigmoid acts as a gating mechanism to compute weights bounded by 0 and 1. Consequently, the fused, multi-representation thermal features are computed by

17

where is the element-wise product. The intuition behind AWFG fusion module is to compute the weighted contribution of the PCT feature maps, , and the fused features, , are the weighted sum of the inputs. The output weights are bounded by 0 and 1 using the sigmoid function, which acts as a gating mechanism for computing the weighted contribution from each representation. Such a mechanism improves the network’s learning since PT-Fusion learns to adaptively compute the contribution of the features instead of a black box concatenation. Once the fused features are evaluated, is fed to the decoder for generating the network predictions.

Algorithm 1.

Adaptive Weighing Fusion Gate (AWFG)

Traditional decoders in U-Net architectures involve skip connections that concatenate the encoded features with the decoder outputs to preserve spatial information extracted by the encoders. However, since PT-Fusion comprises two encoding branches, concatenating the feature vectors from the two encoders with the decoder outputs tends to be memory-intensive. Instead of concatenating all encoded features from both representations, PT-Fusion introduces GEDB fusion module, outlined in Algorithm 2, to adaptively fuse the encoded multi-representation features with the decoder output of the layer . This replaces the skip connection operation in the traditional U-Net. GEDB initially involves passing the features of the m encoder layers through an AWFG block to extract the multi-representation features, , from the m encoder layers of both CNN heads. This is essential to formulate defect-relevant features at different levels of abstraction from all convolutional layers. Then, and are each passed through convolution, and forming and as

18

19

Accordingly, the adaptive gating mechanism outputs learnable weights found by

20

where is a learned convolution. Finally, the output of the decoder to the next decoder layer is the element-wise product of between and as

21

The GEDB operations are repeated until the last decoding layer generates the predicted class logits for segmentation and depth map prediction . Using GEDB, the network dynamically amplifies the important information from the encoders and decoders in a feature vector, enabling accurate defect segmentation along and depth estimation. Instead of relying on blackbox, memory intensive concatenations of encoder and decoder features, GEDB relies on an adaptive gated mechanism to compute the weight of the decoded features, the PCT, and TSR encoded features to generate enhanced segmentation maps. Additionally, computational and memory overheads are reduced since no redundant concatenations are present between the encoder-decoders output.

Algorithm 2.

Gating Enhanced Decoding Block (GEDB)

Data augmentation

As the IRT-PVC dataset comprises 38 inspection sequences, the dataset size is scarce, and training the deep architecture of PT-Fusion remains challenging. While traditional augmentation techniques, such as random rotations, translations, and shearing, introduce spatial variability, they fall short in addressing the spatiotemporal nature of thermographic data. These sequences require augmentations that account for both spatial and temporal dimensions to effectively enhance dataset diversity. Therefore, this work proposes a spatiotemporal augmentation strategy, illustrated in Algorithm 3, to address this limitation. The proposed method involves three steps. First, subsets of thermographic sequences are randomly sampled from the original inspection sequences to introduce temporal variability. Second, additive Gaussian noise is applied on the randomly sampled sequences to further augment the dataset temporally. Finally, PCT and TSR tensors are generated from the randomly sampled sequences as new training samples in the expanded dataset. During training, traditional spatial augmentation techniques are applied dynamically to the generated tensors to enhance the network’s generalizability.

Algorithm 3.

Gating Enhanced Decoding Block (GEDB)

Before augmenting the dataset, 26 sequences out of the 38 in are randomly assigned for training and 6 others for validation. The training and validation datasets are augmented using the proposed augmentation technique in Algorithm 3, while the rest of the dataset is dedicated to testing and remains unaugmented.

For each sequence in the training dataset , where , is divided into equally spaced segments . From each segment, frames are randomly sampled, generating an augmented sequence as

22

where is a uniformly sampled index from the segment , drawn from the segment frame indices. This sequence generation process is repeated with an augmentation factor, , for each sequence. As a result, the dataset currently comprises 16000 samples instead of the original 26. To further expand the diversity of the dataset, additive zero-mean Gaussian noise is added to the augmented sequences by

23

where controls the noise contribution to the sequence mimicking the noise characteristics in thermal imagers. From the augmented sequence , PCT and TSR representations are generated, and traditional augmentation techniques such as random rotation, translation, flipping, and shearing are dynamically applied during training to grant PT-Fusion enhanced generalization. It is important to note that TSR applies logarithmic polynomial fitting which assumes uniform temporal progression. In the proposed data augmentation technique, although frames are subsampled within temporal segments, the reassembled sequence preserves the original temporal ordering. For each selected frame, the corresponding true acquisition timestamp is retained and used during TSR fitting. Thus, the logarithmic mapping in TSR remains consistent with the physical time axis, avoiding distortion of the underlying heat diffusion process. To illustrate, Fig. 7 plots the residuals of the fitted TSR polynomial on the uniformly sampled data against the original data points, highlighting that the residual deviations remain bounded and that the underlying heat diffusion dynamics is preserved.

Fig. 7.

TSR fitted polynomial of two pixels with uniformly sampled data points for data augmentation in (a) and (b), and the corresponding residuals in (c) and (d).

Implementation and training details

After formulating the PT-Fusion architecture incorporating the CNN heads, AWFG, and GEDB with its training data, a Bayesian optimization process over the network is employed to select its hyperparameters. The optimization procedure is initialized with 10 random samples, followed by 40 Bayesian-guided iterations using a Gaussian process surrogate with Expected Improvement acquisition⁴⁸, i.e. 50 trials total. Each trial was trained for 30 epochs with early stopping, resulting in a computational budget of approximately 1,500 training epochs; hence, approximately 25 GPU hours on RTX 4070. In addition, all network weights were initialized with the PyTorch Glorot Uniform initializer. The hyperparameters include the number of convolutional layers, the number of filters, the kernel sizes, and the batch size. During the optimization process, the network weights are initialized using the built-in Glorot Uniform initializer in PyTorch. The resulting optimal network parameters are listed in Table 14. As outlined in Table 14, convolutional layers with kernels are incorporated for the PCT and TSR encoders. This also implies that the 4 GEDBs are employed to fuse the encoded features with the decoders at each m layer.

Table 14.

Search ranges and selected optimal hyperparameters for PT-Fusion.

Hyperparameter	Search Range	Optimal Value
No. Conv. Layers
Conv. Filters
Kernel Size
Dropout Coefficient
Intermediate Loss	Yes / No	No
Batch Size
Epochs
Learning Rate

We train two variants of the network that differ in the final prediction layer. The first predicts a multi-class segmentation mask, segmenting the pixels into defect classes defined by the defect’s depth. For this variant, the training loss function is the cross-entropy loss computed by

24

where N is the number of pixels in the image, is the ground truth, and is the predicted class logits. The second variant of the network comprises two outputs: 1) the binary segmentation mask and 2) the depth map prediction . Hence, the loss function for training the network is a combination of binary cross entropy loss, , for segmentation, and L1 loss, , for depth estimation as

25

where

26

27

is a weighing factor, and is the depth map ground truth. The two variants of the network are trained for a two-fold validation routine. It is worth highlighting that both networks are trained with a batch size of 8 using an ADAM optimizer with a learning rate of , as depicted in Table 14. In addition, both networks are trained from scratch with no frozen parameters on the IRT-PVC dataset augmented using the proposed augmentation technique. This is because it was highlighted in²⁰ that fine-tuning pre-trained a ResNet backbone does not lead to improved training.

Conclusion

Infrared thermography (IRT) became prominent in the field of NDT, enabling the detection of subsurface defects in industrial components. Particularly, pulse thermography (PT) is valued for its efficiency and fast nature. Current state-of-the-art PT inspection networks learn to extract thermal features from PCT and TSR image representations independently for defect segmentation and depth estimation. This limits the performance of the trained networks as these representations possess complementary semantic features. Thus, this work proposes PT-Fusion, an attention-based fusion network that fuses thermal features extracted from PCT and TSR representation for enhanced segmentation and depth estimation of subsurface defects. The network architecture introduces two feature fusion modules, namely, Adaptive Weighing Fusion Gate (AWFG) to adaptively learn the weighted contributions of the input representations and Gating Enhanced Decoding Block (GEDB) to dynamically fuse the encoded features with outputs of the PT-Fusion decoders. In addition, due to the scarcity of PT datasets, a novel spatiotemporal data augmentation technique is proposed to mitigate this challenge and facilitate the training of PT-Fusion. The network is tested on the Université Laval IRT-PVC dataset for defect segmentation and depth estimation. PT-Fusion is also benchmarked against state-of-the-art PT network architectures, such as U-Net, attention U-Net, 3D-CNN, Swin-UNet, and TransUNet. The results demonstrate that PT-Fusion outperforms the aforementioned models with an IoU of 0.882 and a depth estimation error of 0.0082 mm. Moreover, a generalizability study has been conducted to test the generalization capability of PT-Fusion with respect to defect geometries. The obtained results highlighted that the proposed framework is able to generalize to unseen defect geometries and does not overfit to geometric priors during training. Future work will aim at constructing larger and more diverse inspection datasets comprising different materials and capturing a wider range of thermal responses and defect types. Consequently, research efforts will be directed towards extending PT-Fusion to lock-in thermography setups and inspection of other materials such as CFRP and PLA. In addition, experimental studies will be conducted on the efficacy of self and cross-attention mechanisms during the inspection of materials with varying thermal conductivities.

Author contributions

M.S. experiments, methodology, software, writing. N.W. technical supervision, review, editin. D.S. technical supervision, review, editing. S.S. technical supervision, review, editing. Y.A. Project management, technical supervision, and funding acquisition.

Funding

This research was funded by Khalifa University of Science and Technology through the Faculty Start-Up (FSU) Fund under Project ID: KU-INT-FSU-2024-8474000660.

Data availability

The datasets generated during and/or analysed during the current study are available in the PT-Fusion repository, https://github.com/mohammedsalah98/pt_fusion.

Declarations

Competing Interests

The authors declare no competing interests.