PFVnet, a feature enhancement network for low recognition coal and rock images

Abstract

The existing coal-rock identification technology based on machine vision makes it difficult to accurately identify coal-rock images with low distinguishability. To solve this problem, a special coal-rock environment simulation experimental device was used to conduct simulations, considering various influencing factors such as illumination, air flow, coal dust, and water mist concentration. We characterized the grayscale and texture feature patterns of coal-rock media under varying degrees of interference and established a comprehensive multi-element image training sample library. The simulation experiment results show that illumination, dust, and fog can reduce the distinguishability of coal-rock images, which seriously affects the recognition performance of the network. Based on this, the convolution operation was combined with the Vision Transformer network and the deep convolution algorithm was applied to design a parallel hybrid vision network model, PFVnet. Subsequently, enhanced recognition tests were carried out in combination with the DeepLabV3 + network. The test results show that PFVnet can enhance the features of coal and rock, and achieve a PSNR of 18.90 and an SSIM of 0.58 on the multi-element image training sample library. It can effectively reduce the misjudgment of the DeepLabV3 + network, increasing its accuracy by 0.95%, the mean Intersection over Union (mIoU) by 2.15%, and the mean Pixel Accuracy (mPA) by 2.12%. This research provides new ideas and feasible technical solutions for the improvement of coal-rock identification technology and helps to promote the development of this field.

Introduction

In recent years, the coal-rock identification technology based on machine vision has undergone a phase of robust proliferation. Endowed with virtues such as cost-efficiency and technological refinement, this technology has achieved several landmark accomplishments within the realm of coal-rock identification^1–3. Under conditions of mild interference, the characteristic disparities among coal and rock masses are prominently manifest, and the current recognition precision of coal-rock images has ascended to a relatively elevated tier⁴. Nevertheless, when transplanted to field applications, it becomes patently clear that during the on-site mining process, the images of coal-rock masses are subjected to intense perturbations from interfering factors such as illumination and dust-fog. This gives rise to phenomena like the diminution of image brightness and the forfeiture of texture information, thereby precipitating a decline in image recognition accuracy.

An in-depth scrutiny of the underground mining environment reveals that during on-site mining operations, the lighting of the mining face is furnished by stationary light sources. During the shearing process of the coal-cutting machine, the cutting teeth disintegrate the coal blocks, thereby generating coal dust of variable magnitudes. Subsequently, this coal dust interacts with the fine water mist ejected by the coal-cutting machine, leading to condensation and deposition. Throughout the mining process, both non-uniform lighting conditions and residual coal dust and water mist exert an impact on the images, thereby reducing their discriminability. This, in turn, encroaches upon the recognition network and attenuates the recognition accuracy.

To tackle this predicament, it is imperative to address the impacts engendered by interfering conditions and to explore the feature-enhancement methodologies for coal-rock images. From the perspective of the actual influencing factors in on-site mining, enhancement endeavors are requisite in two dimensions, namely, the elimination of dust-fog and illumination rectification. Among these, the fragmented coal dust produced by the coal-cutting machine and the water mist utilized for dust suppression can be conceptualized as a particular type of haze.

In accordance with the processing paradigms of image visibility enhancement, current image de-fogging and illumination correction techniques can be coarsely categorized into three classes: enhancement approaches predicated on image processing, those based on physical models, and those founded on deep learning.

The enhancement approach based on image processing entails the manipulation of local or global image segments in accordance with traditional image processing techniques. For instance, Muniyappan S et al.⁵, Khalid R et al.⁶, and Han J H et al.⁷ directly utilize the histogram equalization method. Through redistribution operations, this method equalizes the grayscale of the image, thereby effectuating illumination correction and image de-fogging operations. Additionally, there exists the dark-channel-prior de-fogging technology predicated on statistical priors⁸. This type of operation customarily hinges on certain prior knowledge as a foundation, devising algorithms to conduct specific operations on the image. It enjoys an advantage in processing speed relative to other methods. However, when confronted with complex and mutable environments, it lacks the necessary generalization capacity, giving rise to adverse visual artifacts such as over-enhancement or information loss.

The enhancement approach based on a physical model mainly processes images by leveraging the designed physical models. Take the Retinex theoretical model of the illumination system as an example^9,10. The Retinex theory, with the human visual system as the fundamental physical model, posits that the brightness S of the original image can be decomposed into a reflection component R and an illumination component L. By performing different operations on the reflection and illumination components, illumination correction of the input image is achieved. Similarly, regarding the basic model of haze, Yang Yang et al.¹¹ and Khmag Asem¹² constructed a degraded imaging model based on the degradation mechanism of foggy images and its elimination theory for reverse restoration. These enhancement methods based on physical models boast strong interpretability. Through meticulous design, they can achieve good enhancement effects. However, as the environmental interference conditions keep increasing, the complexity and computational volume of the models increase exponentially, which affects the processing efficiency of the algorithms.

With the advancement of deep - learning technology, deep learning has achieved good results among researchers due to its powerful fitting and generalization abilities. Ren W et al.¹³ and Li B et al.¹⁴ redesigned the de - fogging task and proposed new de - fogging neural networks, achieving excellent research results in the field of de - fogging. Cai Y et al.¹⁵ and Wu Y et al.¹⁶ redesigned the image enhancement task using deep - learning technology and showed excellent performance in enhancing low - illumination images. However, the concentration levels of coal dust and water mist in underground mines far exceed those of natural haze, and there is also the problem of uneven illumination. Although previous methods are ingenious, they cannot achieve ideal results in enhancing underground coal - rock images, and thus optimization on this basis is required.

For composite tasks involving illumination, texture, de - fogging, and feature enhancement, the enhancement approach based on image processing fails to achieve both local and overall enhancement simultaneously. The enhancement approach based on physical models significantly increases the algorithm’s computational time and the complexity of model design when multiple models are combined. In contrast, deep - learning technology has the characteristics of end - to - end learning and, with data support, can learn deep - level abstract features. It can achieve better performance in the target task of underground coal - rock identification under multi - factor interference^17,18. After comprehensive consideration, this paper extends the research scope to the study of feature enhancement of coal - rock images under interference conditions, exploring the effect of deep - learning technology on enhancing the features of coal - rock images at the mining site to eliminate the influence of interference under compound negative conditions.

Data source

Simulation experiment design

Deep learning requires abundant data samples to extract high-level abstract features from complex data. However, on-site data samples of coal-rock images are unevenly distributed and lack the universality needed for deep learning, failing to fully represent interference factors and increasing training and learning difficulties. To address this, a coal-rock environment simulation experiment device was designed to simulate negative conditions during mining, enabling the study of coal-rock images under interference, determination of image representation rules of different coal-rock media, and construction of a universal multivariate image training sample library.

To avoid external interference, the experiment will be carried out in a closed, dark room within a self-made experimental box. The simulation takes into account multi-dimensional factors like the objective environment, underground illumination conditions, air flow, coal dust, and water mist. The atmospheric conditions in the experimental box remain the same as those in the normal environment. The box (Fig. 1, 500 mm×500 mm×400 mm) has two movable transparent glass plates (front and back) for specimens and camera access, and four 50-mm-diameter holes on each side (50 mm from the side length) for air circulation, coal-dust input, and water-mist dust-suppression.

Fig. 1.

Self-made experimental box.

The experimental process is as follows:

1) Put an appropriate amount of pulverized coal in the coal-dust container in Fig. 2(a) and mix in different-sized coal blocks to simulate mining-induced coal-dust. Store the coal dust in a plastic container and use the blower in Fig. 2(a) to send it into the box, controlling the concentration by adjusting the blower’s air volume.

Fig. 2.

Simulation Experiment Diagram.

2) Use the low-position blower in Fig. 2(b) to make air in the box circulate at a constant speed for roadway ventilation simulation to make the environment closer to actual mining conditions.

3) Place different specimens from the Shuangyang Coal Mine in Jixi, Heilongjiang Province (including coal blocks, rock blocks, and coal-rock complexes) in the box as in Fig. 2(b) and use the small flashlight in Fig. 2(c) for illumination.

4) Start the blower, block the outlet for air circulation, and use a camera to collect pictures of specimens without interference.

5) Release coal dust by lowering the barrier plate and adjust the air volume of the dust-sending device in Fig. 2(a) to obtain pictures of specimens under different coal-dust concentrations.

6) Spray water mist into the box through the inlet in Fig. 2(b) for dust-suppression, reducing dust and creating a dust-and-water-mist environment, then collect coal-rock images.

7) Repeat steps 5 and 6, adjusting relevant parameters to obtain pictures of specimens under different coal-dust and water-mist concentrations.

8) Replace specimens and repeat step 7 to get coal-rock images of different specimens in various states.

Data sample set preparation

After screening, 1,572 photos in 20 groups under different conditions are selected as the data set. According to the interference degree and type, the samples can be divided into four states:

1. Clear images. As shown in Fig. 3(a), clear coal-rock images before sending coal dust are selected as reference samples.

2. Low-concentration coal-dust images. As shown in Fig. 3(b), weakly recognizable coal-rock images under low-concentration coal-dust interference are selected as data samples.

3. High-concentration coal-dust images. As shown in Fig. 3(c), low-recognizable coal-rock images under high-concentration coal-dust interference are selected.

4. Water-mist dust-suppression images. As shown in Fig. 3(d), coal-rock images after water-mist dust-suppression are selected as samples.

Fig. 3.

Example of sample set sampling.

Sample analysis

Analysis of Coal-Rock image samples

For the purpose of obtaining the image representation rules of coal and rock mass images under diverse physical conditions and investigating the disparities in interference levels of these images at the image level, this section undertakes an analysis of the grayscale and texture information within the data samples.

(1) Grayscale and Texture Analysis of Coal and Rock Mass Samples.

As depicted in Fig. 4, isolated coal and rock mass samples are selected for analysis.

Fig. 4.

Coal and Rock Sample Image.

Analyzing the grayscale histograms and texture information of the images, as shown in Fig. 5, the grayscale histogram of the clear coal - block image is relatively rich, while that of the rock block is relatively simple. The main grayscale values are concentrated in the range of 50–150. The information entropy of coal and rock is 7.65 bit and 7.22 bit respectively, indicating relatively abundant information. In terms of texture, the surface of coal is more changeable, while the rock block is relatively smooth. There are significant differences between them in terms of grayscale and texture, so coal and rock can be clearly distinguished from the grayscale histogram and texture detection map.

Fig. 5.

Analysis of Clear Coal and Rock Images.

With the increase of coal dust and water mist, the distinguishability of coal - rock images decreases. As shown in Fig. 6, the grayscale histograms of coal - rock images are concentrated between 25 and 75. The information entropy of coal and rock is 5.85 bit and 5.89 bit respectively. Comparatively, coal is affected to a higher degree than rock, and the loss of information entropy is also higher, being 1.8 bit and 1.33 bit respectively. Compared with the clear state, the differences in grayscale levels and texture between coal and rock have decreased, making it more difficult to accurately distinguish coal - rock images.

Fig. 6.

Analysis of Low-Visibility Coal and Rock Images.

It can be seen that strong interference will damage the information of coal and rock masses at the brightness and texture levels. Specifically, the grayscale distribution is concentrated in low grayscale values, and the texture information is all lost to a certain extent. This makes the characteristic differences between coal and rock masses decrease, affecting the recognition accuracy of coal - rock images. Moreover, the degree of interference on coal is higher than that on rock.

(2) Grayscale and Texture Analysis of Coal-Rock Composite Samples.

Taking a group of coal-rock composite images presented in Fig. 7 as an example, an analysis of coal-rock composite images under four interference levels is conducted.

Fig. 7.

Coal-Rock Composite Sample Images.

a) Clear-state pictures.

As shown in Fig. 8, due to the influence of uneven illumination, there is a strong specular reflection phenomenon in the image, and extreme values exist in the grayscale histogram. The information entropy of the coal - rock mass is 7.61 bit, the texture information and grayscale levels are relatively rich, and the detection map is also relatively clear.

Fig. 8.

Analysis of Clear-State Coal-Rock Composite Images.

b) Coal-rock pictures under low-concentration coal-dust interference.

With the introduction of coal dust, as shown in Fig. 9, due to the diffuse reflection and occlusion of coal dust, the grayscale frequency in the range of 200–255 in the histogram decreases. At the same time, the peak value gradually spreads, which proves that the influence of specular reflection gradually declines, and the overall distribution gradually becomes more uniform. The information entropy of the combination decreases to 7.56 bit, and the texture becomes more disorderly.

Fig. 9.

Analysis of Coal-Rock Composite Images Under Low-Concentration Coal Dust Interference.

c) Coal-rock pictures under high-concentration coal-dust interference.

As shown in Fig. 10, with the increase of coal dust in the air, it can be observed that the high grayscale values from 200 to 255 in the histogram nearly disappear. The specular reflection phenomenon weakens, and the grayscale values are concentrated at two peaks of 25 and 175. The information entropy decreases to 7.27 bit. At the same time, the distinguishability of the surface texture weakens as the amount of coal dust increases.

Fig. 10.

Analysis of Coal-Rock Composite Images Under High-Concentration Coal Dust Interference.

d) Coal-rock pictures after water-mist dust-suppression.

Subsequent to the adsorption and dust-suppression of water mist, the quantity of dust and fog in the picture decreases. As shown in Fig. 11, the grayscale level of the picture rebounds to some extent, and the information entropy increases to 7.55 bit. The texture information of the picture is enhanced, but there is still a gap compared with the coal - rock pictures in a clear state. At the same time, the presence of water mist also adds more interference information, which affects the recognition accuracy of coal and rock.

Fig. 11.

Analysis of Coal-Rock Composite Images After Water Mist Dust Suppression.

From the above analysis, it can be deduced that uneven illumination and dust-fog occlusion exert strong interferences on coal-rock images, resulting in a reduction in the overall grayscale level and a substantial loss of texture information. Moreover, with an increase in coal-dust concentration, the differences in grayscale and texture between coal-rock images gradually decrease, making it more arduous to directly distinguish coal and rock masses in terms of texture and brightness. Even after water-mist dust-suppression, the brightness and texture information of the image are difficult to restore to normal levels.

Recognition analysis of Coal-Rock samples

During the mining process of the shearer, the network is required to accurately recognize coal-rock images for the timely or anticipatory adjustment of the shearer’s height. This entails confronting challenges posed by harsh conditions such as coal-dust water mist and uneven illumination. Experimental findings reveal that the differences between the grayscale values and texture characteristics of coal-rock images decrease with an increase in interference levels, thereby augmenting the complexity of accurately distinguishing coal seams and rocks at the image level.

Taking the DeepLabV3 +¹⁹ network as an instance, it is an image segmentation technology predicated on deep learning, which has achieved numerous accomplishments in the field of image segmentation due to its outstanding feature extraction capabilities. A total of 731 and 432 photos were respectively procured from two mines, and 763 pictures were subsequently selected therefrom for the training and testing of the coal-rock segmentation model. The specific training parameters are presented in Table 1.

Table 1.

Training parameter settings.

DeepLabV3 + Model	Parameter Settings
Batch size	10
Image size	512
Number classes	2
Max epochs	200
Optimizer	Adam
Initial learning rate	5e-4
Weight decay	0
Momentum parameters [beta1,beta2]	[0.9,0.999]

The model was subjected to 200 epochs of training. The resultant metrics are as follows: a pixel accuracy of 95.73; a mean intersection-over-union (mIoU) of 91.43; and a class-average pixel accuracy (mPA) of 95.63.

When testing on clear coal-rock images, the recognition results (as presented in Fig. 12) indicate that although there are a few omission errors in the network due to the image-taking angle and shadow position, both the entire coal seam and the exposed coal seam within the coal gangue can be effectively recognized. The basic accuracy rate of the model has reached 95%.

Fig. 12.

Recognition Results of Clear-State Coal-Rock Images.

However, when performing recognition on low-resolution coal-rock images (as shown in Fig. 13), a substantial number of omission and misclassification errors are present in the network, failing to meet the requirements of intelligent mining. Even if additional image samples under interference conditions are incorporated at the data level and inductive learning is executed using deep-learning technology, the accuracy of high-resolution coal-rock images may be compromised due to data distribution shift, thereby resulting in a counterproductive outcome. The basic accuracy rate of the model has decreased to 91%.

Fig. 13.

Detection Results of Low-Discernibility Coal-Rock Images.

Considering a special case where the interference conditions are extreme (as depicted in Fig. 14), if the quality of the input original image is overly low, it may directly lead to the inability to correctly identify the position of the coal seam, thereby severely affecting the recognition accuracy of coal-rock images.

Fig. 14.

Recognition Results of Low-Resolution Coal-Rock Image.

To enhance the recognition accuracy of the network for low-resolution coal-rock images and to adapt to the complex and harsh underground environment, a coal-rock image feature-enhancement technology is urgently required to improve the image quality.

Feature enhancement methods

PFVnet network model

Previous experiments have shown that under on - site mining conditions, the effects of illumination and dust - fog can cause the loss of grayscale and texture information in coal and rock mass images. This interference undermines the recognition accuracy of coal - rock identification methods, and in severe cases, may even prevent the algorithm from functioning properly. To tackle this problem, this paper proposes a parallel hybrid vision network model named PFVnet (Parallel Fusion Visual Network).

Convolutional neural networks (CNN) take advantage of the inductive bias²⁰ ability of the convolutional layer to extract and combine features from raw data, which allows for the modeling and fitting of complex data. In contrast, Vision Transformer²¹ depends on the parallel global information extraction ability²² of the Transformer²³ architecture, enabling the network to learn multi - level global abstract features. For the coal - rock image feature - enhancement task, convolution is good at extracting local features but has difficulty in obtaining a global representation. Meanwhile, the Transformer structure can capture long - distance feature - dependence relationships, yet it might sacrifice local feature details. To make full use of the advantages of both architectures, the authors in²⁴ proposed a Conformer architecture. It combines the local feature - extraction ability of the convolutional layer with the global representation ability of the Transformer through a parallel structure, maximizing the preservation of local features and global representation.

According to the characteristics of the coal - rock image feature - enhancement task, PFVnet integrates the advantages of convolutional neural networks and the Transformer architecture. Based on the Conformer architecture and the Unet²⁵ architecture, the overall structure of the model is constructed, as shown in Fig. 15.

Fig. 15.

Schematic Diagram of PFVnet Model.

As shown in Fig. 16, the overall architecture of PFVnet consists of three components: the ​​Illumination Enhancement Layer (IE_Layer)​​, a ​​parallel hybrid Encoder-Decoder​​, and a ​​multi-scale feature fusion module​​. It aims to enhance the quality of low-visibility coal-rock images through multi-dimensional feature extraction and dynamic adjustment mechanisms.

Fig. 16.

Encoder-Decoder architecture.

Illumination enhancement layer (IE_Layer)

Neural networks can fit complex nonlinear representations through the combination of linear convolutions and nonlinear activation functions. However, due to limitations in machine performance and model architecture, this fitting ability is constrained. Appropriate specification of priors and network modules with specific functions can facilitate the convergence of neural networks. Owing to the occlusion caused by dust-fog and the reflection of the coal seam, illumination effects can impact the feature enhancement of coal-rock images. To alleviate this impact and accelerate network training, an illumination enhancement layer (IE_Layer) is introduced before the input enters the main network.

The IE_Layer employs two learnable parameters, γ and β, to scale and shift the input channel values, thereby dynamically adjusting the brightness and contrast of the image at the channel level. Let the image pass through the IE_Layer, as expressed in Eq. (1):

1

Where represents the output, and γ and β are learnable parameters. The calculation is performed for each of the three channels separately.

After passing through the IE_Layer, the image undergoes processing and enhancement at the channel level, and the output image then proceeds to the Encoder-Decoder for further learning.

Encoder - Decoder architecture

After passing through the IE_Layer, the input image is enhanced globally. Then, it is necessary to conduct feature learning through the Encoder - Decoder architecture, which is illustrated in Fig. 16. The network processing unit consists of the PMC_layer (Parallel Mixed Convolutional Layer) and the PMAC_layer (Parallel Mixed Attention Convolutional Layer), and the Froward_block is used for fusion. This design aims to effectively utilize multi - scale image information and obtain a more comprehensive representation of image features.

The PMC_Layer and PMAC_layer are presented in Fig. 17:

Fig. 17.

PMC_layer and PMAC_layer.

(1) Parallel Mixed Convolutional Layer (PMC_layer).

Convolution operation is a filtering process characterized by sliding within the region of the kernel size to compute the correlation coefficients of the entire image. Different-sized convolutional layers can be employed to extract diverse local features. In the ConvNeXt²⁶ experiments, it has been verified that a 7 × 7 depth-wise convolution can better extract large-scale features of objects at high resolutions. During the convolution process, the manipulation of increasing and then decreasing the number of channels is also conducive to the network’s learning of different channel information.

Nevertheless, the size of the convolution kernel directly correlates with the number of parameters and computational complexity. A larger convolution kernel can capture extensive local features, but an overly large kernel will augment the number of parameters and computational complexity of the network, resulting in the loss of information regarding small-object targets and the generation of substantial redundant information.

Considering these factors comprehensively, as shown in Fig. 17(a), the PMC layer incorporates two parallel depth-wise convolutional layers of 5 × 5 and 7 × 7 to acquire image feature information at different scales respectively. By emulating the ConvNeXt architecture, the number of channels is initially increased through a fully-connected layer to extract channel information, followed by a decrease in the number of channels for mapping learning. Subsequently, two learnable parameters are utilized to adaptively adjust the proportion of the branches. After Group Norm normalization processing and the combination with the residual branch output, a fused feature map is obtained.

(2) Parallel Mixed Attention Convolutional Layer (PMAC_layer).

During successive down-sampling, convolution, and self-attention operations, the information acquired by the network gradually becomes more abstract, with a greater focus on the global information and deep-layer features of the object. This comes at the expense of sacrificing image details to obtain a global information representation. Although a large convolution kernel can capture more global information, it may cause the network to lose its ability to extract local information. The PMAC layer connects an attention block and a 3 × 3 depth-wise convolutional layer in parallel, thereby capitalizing on both the global information-extraction ability of the attention mechanism and the local information-extraction ability of the convolution to better learn the coal-rock features.

The Transformer network exhibits a powerful capability to extract long-distance sequence information through the self-attention (Self-Attention) mechanism. However, for pixel-based data, the large number of parameters in a single Attention calculation can significantly prolong the training time. To ensure the real-time performance of the model and reduce the time required for a single forward calculation, the PMAC block substitutes the Transformer self-attention block with the FlashAttention2.0²⁷ block. After the input image is down-sampled by a factor of 8, the shallow features obtained by the PMC layer network are subjected to enhanced sequence extraction. Similarly, as depicted in Fig. 17(b), these two branches are connected via learnable branch weights to adaptively adjust the proportion of the branches.

(3) Up-sampling and Down-sampling Layers.

Through the PMC_Layer and PMAC_layer network layers, the network can obtain deep-layer abstract features. However, for the input image, the high resolution often leads to a large amount of redundant information. Down-sampling is thus necessary to reduce this redundancy and accelerate the training speed, followed by up-sampling for image restoration.

In common image segmentation and image restoration tasks, the network typically employs bilinear interpolation or transposed convolution with a stride of 2 for up-sampling the input image. However, simple bilinear interpolation can cause image degradation, while transposed convolution may lead to a checkerboard effect²⁸ due to “uneven overlap”.

To alleviate this problem, in SUNet²⁹, an up-sampling method combining pixel shuffle and bilinear interpolation has been proposed. As illustrated in Fig. 18(a), the input image simultaneously passes through two branches, undergoing bilinear interpolation and pixel shuffle for up-sampling respectively. Subsequently, they are concatenated along the channel dimension through the concat operation, and finally, a convolutional layer is used to fuse the two up-sampled outputs, thereby reducing the occurrence of the checkerboard effect. Adopting a similar approach, the down-sampling layer is designed as shown in Fig. 18(b). In this design, the MaxPool operation retains the parts that respond strongly to the pooling operation, effectively preserving the texture information of the object. The convolution with a stride of 2 can maximally restore the mapping relationship between features. The network simultaneously performs MaxPool and stride-2 convolution on the input image for down-sampling. Subsequently, the results are concatenated and the output features are fused to maximize the retention of features.

Fig. 18.

Mixed Upsampling Layer and Mixed DownSampling Layer.

During successive down-sampling operations, although the network can extract multi-scale information of coal-rock images, each down-sampling step leads to the loss of image detail information and the introduction of additional redundant information. This results in a significant offset between the output image and the original image, which is detrimental to network convergence. To overcome this issue, this paper devises a multi-scale feature fusion Forward Block. As shown in Fig. 19, after the Encoder layer extracts information, the network layer fuses the output feature map with the high-magnification feature map through the Concat operation to obtain a fused representation of multi-scale feature information. Subsequently, the redundant information resulting from the fusion is distilled through the Context block³⁰. Finally, the processed information is sent to the Forward block for enhanced extraction before entering the lower-level Decoder module.

Fig. 19.

Decoder Module.

The Forward layer operates on the input feature map by applying two 1 × 1 convolutions to the channels, followed by depth-wise convolution to enhance feature extraction. After passing through the activation layer and normalization operation, the feature map is then processed by a CBAM mixed-attention module³¹. This module introduces a channel-attention mechanism to the Forward layer, focusing attention on important information and thereby enhancing the feature-extraction ability of the module.

Through the encoder - decoder operations of the network layer, the network can effectively enhance the features of the input coal - rock images. After initial enhancement by the IE_Layer, the images enter the encoder - decoder architecture. In the encoder, the PMC_layer is responsible for multi - scale feature extraction, while the PMAC_layer combines the extraction of both global and local information. The mixed down - sampling process helps reduce redundant information.

Within the decoder, the mixed up - sampling mechanism restores the image details. The Forward Block addresses the problems arising from down - sampling, and the CBAM module in the Forward layer hones in on the crucial features. Through this series of operations, the network achieves enhancement of the coal - rock images in terms of brightness, texture, and other aspects.

Where model training and results

The model training process employs a transfer learning strategy. Initially, training is conducted on the public defogging dataset from NTIRE 2018. Subsequently, based on the model parameters obtained from this initial training, the model is further trained on the dataset derived from the simulation experiment to account for any potential offset of the model on the simulation experiment images.

To enrich the dataset representation and alleviate the risk of overfitting, a data-enhancement strategy is applied to the dataset. Considering the unique characteristics of the target task, the following enhancement techniques are randomly applied to the dataset:

(1) Geometric Transformation: Enhancement is achieved through operations such as rotation, cropping, and flipping. Given that coal is an ore resource with diverse natural occurrences in various shapes and orientations, geometric transformation enhances the model’s adaptability to geometrically variant images.

(2) Color Transformation: Enhancement involves adjustments to brightness, contrast, and hue. During actual mining operations at the working face, illumination is solely provided by fixed light sources. Depending on the type and irradiation angle of these fixed light sources, variations in brightness, contrast, and hue occur. Hence, color transformation augments the model’s capacity to adapt to color variations.

(3) Interference Enhancement: Enhancement is achieved through the introduction of factors such as salt-and-pepper noise, Gaussian blur, and random occlusion. During actual mining, due to the instantaneous occlusion caused by flying coal dust and water mist and the possible presence of random occluders, images are subject to blurring, noise, and occlusion effects. Incorporating such interference factors enhances the model’s adaptability to these disturbances.

After data enhancement, network training is executed using a cloud server equipped with an RTX 3090 (24GB) GPU and a 12-vCPU Intel (R) Xeon (R) Platinum 8255 C CPU @ 2.50 GHz. The learning-rate scheduling adopts the Restart lr training strategy. The detailed parameters are presented in Table 2.

Table 2.

Training information of PFVnet model.

PFVnet Model	Parameter Settings
Batch Size	8
Image Size	192
Input Channel Number	32
Group (Group Norm)	8
Number of PMC Layer Modules	[2,6,6]
Number of PMAC Layer Modules	[2,2]
Number of Heads in Multi-head Attention	[8,16]
Mlp Ratio	4
Number of Forward Modules	[2,2,2,2,2]
Forward Module Channel Expansion Ratio	6
Maximum Iteration Count	100,000
Optimizer	AdamW
Initial Learning Rate	4e-4
Weight Decay	0
Momentum Parameters [beta1,beta2]	[0.9,0.999]
Retraining Learning Periods	[19500,100000]
Starting Weights for Retraining	[1,0.5]
Minimum Learning Rate	[1e-5,1e-6]

To prevent overfitting, the model underwent an initial pre-training on the public dataset for 50 epochs, achieving a best PSNR of 18.52, best SSIM of 0.57, and best MS-SSIM_L1 Loss of 18.622. In the second training phase, the model was trained for 150 epochs, yielding a best PSNR of 18.90, best SSIM of 0.58, and best MS-SSIM_L1 Loss of 17.810.

Evaluation of PFVnet enhanced performance and Coal-Rock recognition accuracy

Feature-Enhancement results based on PFVnet

Given that the average brightness and contrast of the NTIRE 2018 public defogging dataset surpass those of underground coal-rock images, to preclude the occurrence of model-learning deviation, the network first undergoes CLAHE (Contrast-Limited Adaptive Histogram Equalization) histogram equalization. Subsequently, the PFVnet model is employed to conduct feature-enhancement on the coal-rock images acquired from the simulation experiment.

As depicted in Fig. 20, following the histogram-equalization operation, the issue of uneven illumination within the image is ameliorated to a certain degree, accompanied by an elevation in both contrast and brightness. Nevertheless, the image remains subject to the interference of dust and fog, resulting in a rather blurred overall texture, thereby complicating the differentiation between coal and rock. Post-comprehensive enhancement by PFVnet, the textures of coal and rock become distinctly visible, the dust and fog encircling the surface are eradicated, and the luster of the coal seam is augmented, thereby accentuating the disparity within coal-rock images.

Fig. 20.

Comparison of Enhancement Results.

As shown in Fig. 21, the grayscale histogram of the original image is generally concentrated between 25 and 175. There is a relatively small distribution of grayscale values above 200, that is, there is no high - brightness area. The overall texture is difficult to detect, and the information entropy is 7.51 bit. After adaptive histogram equalization, the overall grayscale values are centralized, the high - brightness areas are suppressed, and the overall texture becomes clear. However, the detailed texture is not clear enough, and the information entropy drops to 7.34 bit. After comprehensive processing, the grayscale distribution area of the image becomes rich, the overall texture is clearer, and the information entropy increases to 7.77 bit. Overall, it is enhanced on the basis of retaining the basic characteristics of the coal - rock image.

Fig. 21.

Gray Scale Contrast Chart.

Evidently, the comprehensive PFVnet-enhancement method not only augments the coal-rock image in terms of color and brightness, enriching the color and brightness manifestations of the coal-rock image but also substantially enhances the image’s texture information, thereby magnifying the distinction within coal-rock images.

The same pattern is observable in the samples presented in Fig. 22. Due to the influence of dust, fog, and illumination, the original image presents challenges in detecting clear textures and is characterized by relatively low brightness. After undergoing enhancement via adaptive histogram equalization, the texture is enhanced to a certain extent, yet the detailed texture information is sacrificed, while the image brightness and contrast are enhanced. Post-comprehensive PFVnet-enhancement, the image texture is more distinct, with the detailed texture becoming more prominent, enabling the physical contour to be discerned from the texture.

Fig. 22.

Comparison of Enhanced Texture Detection in Other Samples.

Enhanced recognition results based on PFVnet

The experiment in the preceding section has demonstrated that the DeepLabV3 + model is capable of performing recognition operations on coal-rock images in a pristine state, yielding favorable segmentation results. However, this model is susceptible to interference conditions such as water mist and coal dust, thereby impeding its ability to make accurate distinctions within poor-quality images.To evaluate the enhancement effect of PFVnet on recognition results, coal-rock images, both prior to and subsequent to PFVnet-enhancement, are input into the DeepLabV3 + image-segmentation network. The results are presented in Fig. 23.

Fig. 23.

Comparison of Enhanced Coal Rock Image Recognition.

As depicted in Fig. 23, prior to passing through the enhancement network, the coal-rock image exhibits indistinct feature differences, accompanied by severe misclassification and omission errors within the network. Upon traversing the enhancement network PFVnet, the features of the coal-rock image are fortified, the differences in coal-rock features are magnified, and the misclassification and omission errors are alleviated, thereby enhancing the recognition accuracy of coal-rock image recognition. Additional examples are presented in Fig. 24.

Fig. 24.

Comparison of Enhanced Coal Rock Image Recognition in Other Samples.

Simultaneously, in the extreme example previously described, the network struggles to perform effective recognition on the coal-rock image. Post-PFVnet-enhancement, the recognition results are presented in Fig. 25.

Fig. 25.

Recognition Results of Enhanced Extreme Adverse Images.

It is evident that following PFVnet-enhancement, the features of the coal-rock image are effectively represented, significantly augmenting the recognition accuracy of DeepLabV3 +. This indicates that PFVnet performs commendably even within the harsh recognition environment described previously.

Tests are conducted on the obtained images with varying degrees of interference. The resultant test results prior to enhancement are as follows: pixel accuracy (Accuracy) of 90.7, mean intersection-over-union (mIoU) of 80.79, and class-average pixel accuracy (mPA) of 89.47. Post-enhancement, the test results are pixel accuracy (Accuracy) of 91.65, mean intersection-over-union (mIoU) of 82.94, and class-average pixel accuracy (mPA) of 91.59, all of which exhibit a certain degree of increment. Specifically, the indices of Accuracy, mIoU, and mPA increase by 0.95, 2.15, and 2.12 respectively. This attests to the fact that PFVnet can effectively enhance the features of coal-rock images, rendering these features more distinct and thereby enhancing the image-recognition accuracy. Notably, on a mobile personal computer equipped with a GPU of 3070ti, the average time for a single enhancement operation on an image with an input resolution of 1440 × 1440 is 18.56s, and the average time for the recognition operation is 1.59s.

Discussion

Coal - rock recognition technology is of utmost importance as it is one of the key technologies in the construction of intelligent coal mines. It plays a significant role in enhancing the efficiency and accuracy of coal seam detection, mining, and sorting. Experimental results demonstrate that PFVnet can enhance the features of coal - rock, improve the recognition accuracy of low - resolution images, and is conducive to on - site application. Compared with traditional methods, the technology based on PFVnet has remarkable advantages in terms of effectiveness and generalization ability. However, it has deficiencies in real - time performance and data requirements, leaving room for optimization.

The learning of neural networks depends on data quality, and abundant on - site data is crucial for their learning ability. However, the coal - rock data at the mining site is severely interfered with, resulting in a single - type of images and uncontrollable interference conditions, which brings difficulties to the research. We obtained data through a self - made simulation device. Although the simulation has limitations, the results show that the feature - enhancement method based on deep learning can improve the recognition accuracy of low - resolution images. In the subsequent research, further optimizing the simulation to make it closer to the actual mining conditions can better verify its on - site application effect and promote the research on intelligent mining.

Using the simulated dataset, we adopted the supervised learning method and designed tasks centered around image enhancement. We designed a network to enhance the features of low - resolution coal - rock images and compared them with clear images to optimize the results. During the experiments, it was found that the network performs well in enhancing images with weak interference. However, for images with complex interference, the enhancement effect may decline due to the shift in data distribution. The optimization of neural networks is global, which endows them with generalization ability. We adopted a transfer - learning strategy: first learn the data patterns of coal - rock images with weak interference, and then relearn for those with strong interference. The advantage of this strategy is that the optimization direction is controllable, but it may reduce the final accuracy and has certain limitations. Optimizing this strategy can improve the model performance and task effectiveness.

We envisioned a comprehensive coal - rock recognition task, adopting a strategy of enhancing first and then recognizing. The enhancement network can improve the texture and brightness, making the differences between coal and rock more obvious. However, as a multi - stage task, the strategy of enhancing first and then recognizing may lead to a waste of network computing resources. In future research, integrating the enhancement network as a module into the overall recognition network can further enhance the real - time performance and accuracy of the computation and reduce the computing and data requirements.

In essence, coal - rock recognition is affected by illumination and dust - fog, and our method can mitigate these effects. For on - site mining, optimizing the existing technology by combining appropriate lighting and dust - reduction measures, and supplemented with the enhancement network, can meet the accuracy requirements of coal - rock recognition for intelligent mining and get closer to on - site application.

Summary

We employed a self - made coal - rock environment simulation device for experiments, factoring in multiple variables like illumination, air flow, and dust - mist concentrations. This approach overcame the limitation of single - type on - site data, generating a diverse data sample set of coal - rock images under complex interferences. Analysis of this set showed that illumination and dust - fog degrade coal - rock image recognizability in brightness and texture, negatively affecting network recognition.

To address this, we designed PFVnet, a parallel hybrid vision network based on Unet and Conformer architectures. It aims to enhance input coal - rock image features. On the mixed dataset, it achieved a PSNR of 18.90, an SSIM of 0.58, and an MS - SSIM_L1 Loss of 17.81. By integrating the adaptive histogram equalization algorithm, we enhanced low - resolution images, boosting their feature recognizability. This comprehensive algorithm enriches grayscale levels, sharpens texture features, lessens the impact of dust - fog and uneven illumination, and accentuates coal - rock distinctions.

To assess PFVnet’s impact on coal - rock recognition accuracy, we conducted a refined test on the DeepLabV3 + network. Results demonstrate that PFVnet enhances coal - rock features in brightness and texture, minimizing misclassification and omission. On the low - resolution dataset, it attained the same PSNR of 18.90 and SSIM of 0.58. Moreover, it improved the DeepLabV3 + network’s performance on low - resolution images, with an Accuracy increase of 0.95%, an mIoU rise of 2.15%, and an mPA growth of 2.12%. These improvements suggest that PFVnet holds great potential for practical application in coal - rock recognition systems in coal mines, facilitating more efficient and accurate coal mining operations.

Author contributions

Cai, Han: Planning, Investigations, Experiments, Discussion, Writing major part. Zhenwei, Liu; Shenglei, Zhao: Resources, Co-Supervision, Discussion and Writing. Yubo, li; Yanwei LI; Xinzhou, Yang: Co-Supervision, Discussion and Writing. Chuanbo Hao: Planning, Resources, Supervision, Project administration, Discussion and Writing.

Data availability

The datasets used and analysed during the current study are available in the NTIRE 2018 public dehazing datasets repository, https://data.vision.ee.ethz.ch/cvl/ntire18//o-haze/ and https://data.vision.ee.ethz.ch/cvl/ntire18//i-haze/. The datasets generated during the current study through simulation experiments related to our research results are not publicly available due to the privacy concerns of our research team but are available from the corresponding author on reasonable request. The corresponding author can be reached at hancai1999@outlook.com.

Declarations

Competing interests

The authors declare no competing interests.