A hybrid local-global feature attention network for thin section rock image classification

Abstract

Accurate classification of thin-section rock images is critical for geological surveys, resource exploration, and automated petrographic analysis. However, existing deep learning models struggle to balance local texture representation and global semantic context, often leading to suboptimal performance in fine-grained rock type recognition. In this study, we propose HFANet, a novel hybrid local–global feature attention network that integrates a DenseNet-based local branch and a Swin Transformer-based global branch to simultaneously capture detailed textures and contextual semantics. A multi-head self-attention module enhances local feature interactions, while a bidirectional cross-attention mechanism enables dynamic guidance between local and global representations. To further improve discriminative capability, we introduce an ensemble classification framework comprising three independent heads with adaptive fusion, and incorporate handcrafted geological features—such as color, texture, and luminance—via multimodal fusion. Extensive experiments on a comprehensive rock thin-section dataset demonstrate that HFANet outperforms a wide range of state-of-the-art models across multiple tasks, achieving up to 99.03% accuracy and perfect AUC and AUPR scores on the sedimentary subset. Ablation studies and interpretability analyses confirm the effectiveness of each component and the model’s ability to focus on geologically meaningful regions. The results highlight HFANet’s strong potential for advancing intelligent geoscientific image analysis and facilitating practical applications in lithological classification.

Introduction

Rock classification is a cornerstone of geoscience with broad applications in energy exploration and resource development^1–4. For example, as a typical reservoir rock, its accurate interpretation and classification from microscopic images provide crucial support for stratigraphic analysis, resource potential evaluation, and development planning¹. In geological surveys, engineering exploration, and mineral prospecting, reliable rock identification is essential for understanding stratigraphic structures and optimizing resource utilization strategies^2,5. In mining engineering, precise acquisition of rock physical parameters and mineral composition plays a key role in guiding economically viable extraction and ensuring safety and efficiency^6,7. As a result, microscopic rock analysis has become an indispensable tool for geologists and engineers in both research and practical applications⁸.

Traditional rock image analysis relies heavily on hand-crafted features, which require extensive preprocessing and depend strongly on the operator’s experience, leading to unstable performance^9–11. These features typically capture only low-level information such as color, texture, and brightness, making it difficult to represent the complex structures present in rock images. Additionally, due to scale variations and high intra-class similarity, hand-crafted features lack the generalization ability needed to handle large-scale and diverse rock datasets^1,5. Therefore, developing automated methods for accurate parsing and classification of rock images is essential for advancing geoscientific research and supporting efficient resource development^1,12.

Deep learning, as a core technology in artificial intelligence, has achieved remarkable progress in computer vision in recent years¹³. Its key strength lies in automatically learning features from data¹⁴, eliminating the limitations of traditional manual feature extraction and enabling end-to-end task processing¹⁵. Various network architectures have played transformative roles in image classification. Early convolutional neural networks (CNNs), such as LeNet¹⁶, demonstrated potential through convolution and pooling, but adoption was limited by data and computational constraints. AlexNet¹³ initiated a breakthrough in the 2012 ImageNet Challenge with its deep architecture and GPU acceleration, sparking the deep learning revolution. Subsequent models, including VGGNet¹⁷, GoogLeNet¹⁸, and ResNet¹⁹, introduced innovations such as deeper layers, multi-scale Inception modules, and residual connections, achieving significant improvements in accuracy and efficiency. DenseNet²⁰ and EfficientNet²¹ further enhanced feature reuse and model scaling strategies. More recently, Transformer-based models have reshaped the landscape: Vision Trans-former (ViT)²² introduced patch-based self-attention mechanisms suitable for large-scale image data, while Swin Transformer²³ reduced computational costs through hierarchical window attention, promoting widespread adoption of Transformers in vision tasks.

In recent years, numerous studies have applied deep learning techniques to rock image classification, yielding significant results. Pham and Shin developed a model based on ResNet, achieving 84% accuracy on a dataset comprising ten rock categories²⁴. Baraboshkin et al. employed GoogLeNet and ResNet for automated core analysis, attaining 95% precision with a processing speed of 50 m per minute²⁵. Fan et al. proposed a lightweight model based on ShuffleNet and transfer learning, enabling real-time recognition with over 95% accuracy on mobile devices²⁶. Alzubaidi et al. utilized the ResNeXt-50 architecture to classify sandstone, limestone, and shale, reaching an accuracy of 93.12%²⁷. Guojian and Peisong applied ResNet50 and ResNet101 for fine-grained classification of thin-section images from the Ordos Basin, achieving 90.24% and 91.63% accuracy, respectively²⁸. The application of Transformer architectures has also proven effective; Koeshidayatullah et al. introduced FaciesViT, a Vision Transformer-based model which achieved 95% accuracy under limited data conditions²⁵, and Wang et al. proposed Rock-ViT, incorporating supervised contrastive loss to enhance recognition²⁹. Furthermore, various works have focused on improving specific architectures and fusion strategies. Huo et al. improved the Swin Transformer by integrating SAEM, LPB, and BAM modules¹. Zhu et al. developed RockS2Net, which fuses local and global features using STN and dilated convolution⁵, while Zhang et al. proposed SGCT-DenseNet, incorporating GCT and SoftPool³⁰. Other approaches include integrating traditional methods, such as the PCA-VGG16 model by Zhang et al.³¹, and exploring multi-modal data, as demonstrated by Ma et al. with a dual-modal fusion model for polarized images³². Building on this foundation, more recent studies continue to push the boundaries. Bai et al. proposed a lightweight EfficientNet-B0-based model integrating DiffuseMix data augmentation and Shuffle Attention³³. Zhao et al. developed TLCA-ResNet34, combining transfer learning with Coordinate Attention to capture directional textures³⁴. Meanwhile, Dong et al. introduced SENet-ConvNeXt-FL, using Focal Loss to mitigate class imbalance³⁵. Concurrently, innovations in broader geoscientific remote sensing offer valuable insights, such as the SFA-Net by Gao et al., which uses SAM to guide a focused attention network for multimodal image matching³⁶, and another work by Gao et al. that embeds physical model priors into a Transformer’s attention mechanism for sea ice motion retrieval³⁷. Notably, Zhu et al. proposed the Global-Detail SR (GDSR) network, which synergistically unifies RWKV and CNNs in a dual-branch framework to capture both global contextual dependencies and localized spatial features, utilizing a Global-Detail Reconstruction Module to bridge these complementary roles³⁸. Recent advances in multi-modal image fusion and remote sensing further validate dual-branch architectures with frequency-domain analysis. Zhang et al. proposed FSCMF, integrating CNN and Transformer with DCT for infrared-visible fusion³⁹, and STWANet, combining wavelet transform with spatio-temporal attention for change detection⁴⁰. These cross-domain approaches demonstrate the effectiveness of hybrid frequency-spatial designs, providing additional motivation for our architecture. Collectively, these studies highlight a clear trend towards more intricate feature fusion and attention-based architectures.

Although deep learning approaches have achieved impressive results, a critical limitation remains in effectively balancing local and global feature extraction. To bridge this gap, recent works have proposed various fusion strategies. For instance, RockS2Net⁵ employs a Siamese network where one branch processes the full image and another processes a “critical area” located by a Spatial Transformer Network (STN). While innovative, its fusion mechanism is fundamentally shallow, as it only averages the final classification scores. This means the local and global feature representations are extracted in isolation, preventing any deep, mutual guidance or refinement during the learning process. On another front, high-performing models like SGCT-DenseNet³⁰ demonstrate the power of enhancing a single-branch CNN architecture with sophisticated internal modules like Gated Channel Transformation (GCT) and SoftPool. However, as a pure CNN-based solution, it inherently lacks a dedicated architectural component for explicitly and powerfully modeling long-range spatial dependencies, a task for which Transformer architectures are purpose-built. These approaches highlight a persistent gap in the literature. Consequently, a more fundamental challenge remains: how to design an architecture where truly heterogeneous local (CNN-based) and global (Transformer-based) feature representations can deeply and bidirectionally interact, allowing each to guide and refine the other throughout the feature learning process. The absence of such a deep, interactive fusion mechanism leads to several downstream problems: models struggle to focus on the most relevant regions across multiple scales simultaneously, high visual similarity between certain rock types (e.g., quartz sandstone and feldspathic quartz sandstone) continues to cause classification confusion, and the inability to effectively integrate features spanning from microscopic grains to macroscopic textures hinders a comprehensive representation of complex geological structures.

To address the limitations of previous methods, we propose HFANet, a dual-branch network that captures both global and local features through distinct underlying mechanisms. The global branch, based on a pre-trained Swin Transformer, models the image’s overall structure. It achieves this via a shifted window attention mechanism, which builds a global context by progressively establishing connections between different image regions. This attention-based approach excels at modeling long-range spatial relationships. Conversely, the local branch focuses on fine-grained details by applying a pre-trained DenseNet to distinct image patches. Within each patch, deep convolutional operations are performed. Convolution, by its nature, uses localized receptive fields to effectively extract low-level features like textures and edges. This complementary design—global context from cross-window attention and local detail from intra-patch convolution—directly mitigates the weakness of convolutional networks in modeling global context and the limited local sensitivity of Transformer-based models. HFANet facilitates a sophisticated interaction between its global and local feature branches through a dedicated module. This process begins with a multi-head self-attention mechanism applied exclusively to the local features, which serves to enhance their internal structural representations before they are integrated with the global context. Following this initial refinement, a bidirectional multi-head cross-attention module enables mutual guidance and reinforcement between the two streams. In this module, each stream takes turns acting as the query to attend to the other. Specifically, the overarching global features provide high-level contextual guidance to the local features, steering the model’s focus towards semantically important regions. Concurrently, the enhanced local features inject fine-grained details back into the coarse global representation by inverting these roles. This reciprocal mechanism of guidance and enrichment creates a powerful and effective fusion, reinforcing attention on critical areas across different scales and thereby significantly improving the model’s multi-scale feature integration capabilities. The model also includes three independent classifier heads: one based on global features guided by local features, and two based on local features guided by global features—processed through max pooling and average pooling, respectively. These heads encourage diverse learning perspectives, and their outputs are fused through a learned weighting strategy using a linear layer that adaptively adjusts the contribution of each branch. This ensemble classification framework enhances the model’s robustness and effectively addresses the problem of class confusion caused by high visual similarity among certain rock types.

We extensively evaluate HFANet on a comprehensive thin-section rock image dataset, which includes sedimentary, igneous, metamorphic, and mixed-type rock samples. The proposed model is compared against a range of widely adopted computer vision architectures, including ResNet, EfficientNet, DenseNet, VGG, Vision Trans-former, Swin Transformer, and ConvNeXt, all initialized with ImageNet pre-trained weights⁴¹. Experimental results show that HFANet consistently outperforms these baselines across multiple metrics, including accuracy, macro-averaged F1 score, and AUC. Furthermore, we explore the integration of handcrafted geological features—such as color, texture, and luminance—into the global branch via a multimodal fusion module. This fusion leads to additional improvements in classification performance, particularly in fine-grained settings with high inter-class similarity. Ablation studies and interpretability analyses further validate the contribution of each architectural component and demonstrate the model’s ability to attend to geologically meaningful regions. Moreover, the main contributions of this work are summarized as follows:

1. We propose HFANet, a novel hybrid dual-branch architecture that combines a Swin Transformer-based global branch and a DenseNet-based local branch to capture both semantic context and fine-grained textures in thin-section rock images. And we introduce a bidirectional multi-head cross-attention mechanism that enables deep interaction between local and global features, enhancing focus on discriminative regions and improving multi-scale feature fusion.

2. We design an ensemble classification framework composed of three independently guided classification heads with adaptive fusion, which improves robustness and mitigates misclassification caused by high inter-class visual similarity.

3. We conduct extensive experiments across sedimentary, igneous, metamorphic, and mixed rock datasets, including ablation studies, interpretability analysis, and feature fusion validation, demonstrating the generalization, transparency, and superiority of HFANet over a wide range of baseline models.

Materials and methods

Data description

In this work, we employ the photomicrograph dataset of rock thin sections developed by Nanjing University⁴² for developing and testing our model, and an independent mineral thin section dataset⁴³ for cross-dataset generalization validation. The dataset is a comprehensive and high-quality collection designed to support geological teaching and related research. Based on long-term accumulation of educational resources, the dataset includes 108 representative rock types—28 sedimentary, 40 igneous, and 40 metamorphic. A total of 324 thin sections were prepared and photographed under polarized light, resulting in 2,634 images. Each sample comes with detailed identification data, and the images include both plane-polarized and cross-polarized views taken at various angles. This standardized dataset is not only suitable for teaching, but also holds value for machine learning applications and automated rock identification.A representative example of each of the three major rock types included in the dataset is shown in Fig. 1. The following sections introduce the primary dataset’s three subsets and the independent generalization test dataset in detail.

Fig. 1.

The three major rock types in the photomicrograph dataset.

The following sections introduce three data subsets in detail.To provide a clear overview of the dataset’s composition and address potential class imbalance issues, the distribution of samples is visualized in Fig. 2. As shown in Fig. 2a, the dataset is relatively balanced at the level of the three main rock types, comprising 939 igneous, 926 metamorphic, and 657 sedimentary samples. However, a significant class imbalance is evident within the subtypes of each major category. For instance, in the sedimentary subset (Fig. 2b), the number of samples ranges from 168 for Sandstone down to just 24 for several other types like Siltstone and Bauxite. Similar imbalances are observed in the igneous (Fig. 2c) and metamorphic (Fig. 2d) subsets, where certain classes (e.g., Acidic Rock, Schist) are far more represented than others. This class imbalance poses a challenge for classification models, as they may become biased towards the majority classes. To address this class imbalance and prevent model bias towards majority classes, we employ an undersampling strategy during training that downsamples over-represented classes based on the median sample count across all subtypes, thereby significantly reducing the disparity between majority and minority class distributions. The specific implementation details of this approach are provided in the “Experiment Setup” section. Additionally, our experimental setup emphasizes evaluation metrics that are robust to class imbalance, as detailed in the same section. The following sections introduce three data subsets in detail.

Fig. 2.

Distribution of samples in the dataset. a Overall distribution of the three main rock types. b–d Distribution of samples across subtypes for Sedimentary, Igneous, and Metamorphic rocks, respectively, highlighting the class imbalance within each subset.

Sedimentary rock subset

This subset contains 84 thin sections of 28 sedimentary rock types, with 699 polarized micrographs in total. The collection covers a wide range of common sedimentary lithologies, such as pyroclastic rocks, various sandstones, mudstones, siltstones, limestones, dolomites, siliceous rocks, evaporites, and ore-bearing sedimentary rocks. Descriptions of each thin section follow established criteria, including rock naming, classification, composition, and texture. Representative polarized-light images of selected sedimentary rock subtypes, such as tuff, siltstone, and sandstones, are presented in Fig. 3 to illustrate the diversity and detail of this subset.

Fig. 3.

The three subtypes of the sedimentary rock in the subset.

Igneous rock subset

The igneous rock subset includes 120 thin sections from 40 igneous rock types, producing a total of 963 images. It spans the major compositional groups: ultramafic, mafic, intermediate, felsic, and alkaline rocks. Representative rock types include basalt, granite, diorite, rhyolite, among others. The images capture key igneous textures and structures, such as porphyritic, vesicular, and flow banding features. Figure 4 displays typical micrographs of several igneous rock subtypes, including ultramafic, intermediate, and alkaline rocks, showcasing the wide compositional and textural range covered by this dataset.

Fig. 4.

The three subtypes of the igneous rock in the subset.

Metamorphic rock subset

This subset comprises 120 thin sections representing 40 types of metamorphic rocks, with a total of 972 images. It covers 17 major rock categories, including mylonites, schists, gneisses, hornfelses, marbles, serpentinites, and others. A range of metamorphic minerals can be observed, such as sillimanite, andalusite, serpentine, and garnet. The thin sections also illustrate common textures and structures, including recrystallization, foliation, and deformation features. To highlight the structural complexity and mineralogical variety of the metamorphic rock subset, Fig. 5 presents representative thin section images of schists, gneisses, and hornfelses.

Fig. 5.

The three subtypes of the metamorphic rock in the subset.

Mineral thin section dataset

This mineral thin section dataset contains thin section micrographs of 11 mineral types, with 1446 polarized microscopic images in total. The collection encompasses key rock-forming minerals (olivine, augite, hornblende, biotite, plagioclase), typical metamorphic minerals (andalusite, cordierite, staurolite, garnet, actinolite), and sedimentary structures (oolites), representing the three major rock categories. Images were acquired using a Leica DM2700P polarizing microscope at 50× magnification under both plane-polarized and cross-polarized light conditions to capture the diverse optical properties of each mineral type. The dataset follows established mineralogical identification criteria, with careful selection of fresh, well-formed mineral grains that display characteristic optical features across multiple orientations. Representative microscopic images of selected mineral types, such as olivine, staurolite, and oolites, demonstrate the optical diversity and diagnostic features captured in this comprehensive mineral identification dataset.

Manual feature extraction

Color feature

Color features are critical for distinguishing different rock types in image classification tasks. We extract two types of color descriptors: statistical features and PCA-based features. The statistical features include the mean, standard deviation, variance, skewness, and kurtosis, calculated for each RGB channel, resulting in a 15-dimensional vector. These metrics capture the global color distribution and variation patterns. The calculation formula are as follows:

1

2

3

4

where denotes the pixel value at position i in the color channel ; N is the total number of pixels in the image; is the mean value of channel c; is the standard deviation; represents the skewness, and represents the kurtosis. In addition, PCA⁴⁴ is applied to the RGB channels to extract the main directions of color variation while reducing dimensionality. Each image is first mean-centered, followed by eigen-decomposition of the covariance matrix, and the top n principal components are selected to form a 3n-dimensional feature vector. By combining both statistical and PCA color features, we obtain a comprehensive representation of color characteristics. The former reflects overall distribution properties such as contrast and balance, while the latter captures dominant variation trends, offering complementary information for rock image classification.

Texture feature

Texture features are essential for describing surface details in rock thin-section images. We extract four types of texture descriptors: Gray-Level Co-occurrence Matrix (GLCM)⁴⁴, Local Binary Pattern (LBP)⁴⁵, Discrete Wavelet Transform (DWT)⁴⁶, and Fast Fourier Transform (FFT)⁴⁷.

GLCM captures spatial relationships between gray levels and computes statistical metrics such as contrast, correlation, energy, and entropy. They are calculated as follows:

5

6

7

8

where, represent the gray-level values of the pixels, represents the gray co-occurrence matrix, , are the mean and standard deviation of gray level i respectively, and is the minimum value to prevent numerical errors. LBP encodes local texture patterns by thresholding neighborhood pixels and forming histograms that reflect micro-structural texture. Its value is calculated as follows:

9

where P represents the number of sampling points in the neighborhood, R is the sampling radius, is the pixel value in the neighborhood, and is the pixel value in the center. DWT decomposes images into frequency subbands at multiple scales (LL, LH, HL, HH), allowing statistical analysis of both coarse and fine textures. They are calculated as follows:

10

11

where is the coefficient value after wavelet transformation, and N is the number of subband coefficients. The low frequency subbands (LL) retain the overall structural information of the image, while the high frequency subbands (LH, HL, HH) reveal the texture details of the rock surface. FFT transforms images into the frequency domain and extracts global periodic features by computing spectral mean, variance, energy, and entropy. The calculation formulas are shown below:

12

13

where, H and W represents the height and width of the image, respectively; represents the pixel value at the coordinate in the space domain; represents the transformation result of the image in the frequency domain, which is called the spectrum; u, v, represent coordinates in the frequency domain, corresponding to the frequency components of the image in the horizontal and vertical directions; j represents the imaginary unit. Fourier features can distinguish between regular and disordered textures of rocks, which enables the model to effectively learn different expressions of rock surface morphology.

These descriptors collectively characterize both local and global texture properties. GLCM and LBP highlight fine structural patterns, DWT supports multi-scale texture analysis, and FFT captures global frequency information, providing complementary cues for rock classification.

Luminance feature

In the task of rock thin section image classification, brightness features are essential for capturing both the overall illumination characteristics and local structural variations within the images. To comprehensively represent brightness information, we extract three types of features: statistical metrics, gradient-based features⁴⁸, and Laplacian energy⁴⁹. The statistical metrics include the mean, standard deviation, variance, skewness, and kurtosis of pixel intensity values, which collectively reflect the general brightness level and distribution patterns. Gradient-based features are computed using Sobel operators along horizontal and vertical directions to quantify spatial changes in brightness, effectively highlighting texture transitions and edge regions. The calculation formula is as follow:

14

where I represents the image intensity function, and are the gradients in the horizontal and vertical directions respectively, and G is the gradient amplitude. The Laplacian energy is derived by applying a Laplacian filter to the image and calculating the variance of the response, which reflects the sharpness of edges and the degree of local brightness fluctuation. The following formulas are used for the calculation:

15

16

where L is the Laplacian operator and reflects the intensity of brightness change in the edge region of the image. Together, these features provide a comprehensive representation of brightness information, enhancing the model’s capacity to distinguish subtle visual differences between rock types.

HFANet model

HFANet introduces an innovative framework integrating a dual-branch architecture with multi-mechanism strategies for efficient multi-scale feature learning and classification. The architecture of the model is illustrated in Fig. 6. The global branch leverages Swin Transformer to capture holistic contextual information, while the local branch employs DenseNet for fine-grained feature modeling of partitioned image patches, addressing the limitations of CNNs and Transformers in feature representation. A multi-head self-attention mechanism enhances local feature interactions through mutual information computation, and a bidirectional cross-attention module facilitates mutual guidance and fusion between global and local features, improving focus on critical regions and alleviating multi-scale integration challenges. The model adopts a multi-perspective classification approach using three independent heads to process global-guided, max-pooled local, and average-pooled local features, with a learnable linear layer dynamically fusing outputs for final predictions. This ensemble strategy enhances discriminative performance on complex class boundaries. The following sections delve into the technical details of these core components.

Fig. 6.

The architecture of HFANet model.

Dual branch structure

The HFANet model uses a dual branch structure to extract global and local features respectively, and the global branch uses the pre-trained Swin transformer model for feature extraction. Swin Transformer is a deep learning model based on the Transformer architecture, which can efficiently extract global features by using the “Shifted Swin Transformer is a deep learning model based on the Transformer architecture, which can efficiently extract the global features of an image through the “Shifted Window” mechanism and hierarchical structure design. Its “Shifted Window” mechanism performs self-attention computation within a localized window²³. The calculation formula is as follows:

17

Where Q represents the Query matrix, which contains a set of queries for each element in the input sequence; K represents the Key matrix, which contains a set of keys for each element in the input sequence; V is the Value matrix, which contains the information or content for each element in the input sequence.

The global information in the image is also integrated by capturing long distance dependencies across windows through window shift operations. The shift operation can be represented as follows:

18

where represents the shifted feature map resulting form the window shift operation; X represents the input feature map before the shift; () are the spatial indices of a feature within the map; is the channel index of the feature; M is the window size used for the attention computation; H and W are the total height and width of the input feature map.

In addition, its hierarchical structure gradually extracts features from local to global, and the high-level features incorporate rich semantic information with powerful glob-al characterization capabilities, providing high-quality global feature descriptions for subsequent tasks.

Local branching first divides the image into small chunks of set dimensions, and then both the length and width of the image chunks are adjusted to 224 using bilinear interpolation⁵⁰ to prepare for subsequent input to the pre-trained model. Bilinear interpolation calculates the pixel values at the target location by linearly interpolating between the four nearest neighboring pixel points, which is computed using the following formula:

19

20

21

In theses equation, x and y are the coordinates of the points at which we want to estimate the function value, while , , , are the coordinates of the points at which the function value is known. Bilinear interpolation is computationally relatively simple and provides a good balance between computational efficiency and interpolation quality. Then the new normalization operation of the color of the image block is used to avoid the impact of the interpolation operation on the picture, which is calculated under the formula:

22

where is an input feature value in a specific channel c of an image block; is the mean of all feature values within that specific channel c; is the standard deviation of all feature values within that specific channel c.

Then, the features of each image block are extracted using the pre-trained DenseNet model²⁰, an efficient deep convolutional neural network that enhances feature transfer and reuse through its unique dense connectivity mechanism, where each layer is connected to all previous layers. This design not only effectively mitigates the gradient vanishing problem, but also enables the network to efficiently extract rich local features. In the DenseNet model, the densely connected layers gradually extract features through multiple convolutional layers, and the output of each layer is directly received and utilized by the subsequent layers, resulting in a multilevel representation of local information. The mathematical expression for dense connectivity is as follows:

23

24

where is the output feature map of the l-th layer; represents the composite function of operations within the l-th layer (e.g., Batch Normalization, ReLU, Convolution); represents the concatenation of the output feature maps from all preceding layers (from layer 1 to ).

Meanwhile, the transition layer further compresses and fuses key local features while reducing the feature dimension. This structure enables the DenseNet model to efficiently capture local details in an image and generate feature representations with high expressive power, which is well suited for tasks that require attention to local features.

Multi-head self-attention mechanism

The multi-head self-attention mechanism is a core component in the Transformer architecture, which realizes multi-angle and multi-level feature extraction of the input data by decomposing the input data into multiple different “heads”, calculating the self-attention separately, and then stitching the results of these heads together and performing a linear transformation⁵¹. This mechanism is able to capture the long-range dependencies between different locations and learn the feature representations of different subspaces in the data through the parallel computation of multiple heads, which significantly enhances the model’s ability to understand complex data structures. The mathematical expression of the multi-head self-attention mechanism is as follows:

25

where X is the input feature matrix to the multi-head attention block; h is the total number of parallel attention heads; is the output of the i-th scaled dot-product attention head; Concat is the concatenation operation that joins the output matrices from all h attention heads along the feature dimension; is the output projection weight matrix, which is a learned linear transformation applied to the concatenated outputs to produce the final result.

This mechanism can not only help the model better understand the interactions between local features, but also solve the problem of insufficient attention to key regions in traditional methods to a certain extent. Through the parallel computation of multi-head self-attention, the model can pay attention to multiple local regions at the same time and dynamically assign more weights to the key features, thus improving the sensitivity to important information. In addition, the flexibility of the multi-head self-attention mechanism allows the model to capture long-range dependencies of local features at different levels, which further enhances the model’s ability to under-stand and model local features.

Bidirectional cross-attention mechanism

Cross-attention mechanism is an attention mechanism used to deal with multi-modal data or interactions between different sequences⁵¹. Unlike self-attention, cross-attention involves two different input sequences, one called the query sequence and the other called the key and value sequences. Cross-attention dynamically extracts relevant information from the value sequence by calculating the similarity between the query sequence and the key sequence, thus realizing the information interaction and fusion between the two sequences. This mechanism can effectively capture the complex dependencies between different sequences and enhance the model’s ability to understand cross-sequence information.

In order to further enhance the model’s attention to key regions and fuse multi-scale information more effectively, we propose a two-way multi-head cross-attention mechanism. The core idea of this mechanism is to allow global and local features to be used as queries to compute multi-headed cross-attention with the corresponding another feature (i.e., local feature or global feature), respectively. Specifically, the first step is to use the local feature as a query to extract the most relevant information from the global feature to enhance the global perception of the local feature, while the second step is the reverse computation, i.e., to use the global feature as a query to extract the key information from the local feature that contributes to the overall representation. Through this bi-directional computation, dynamic information exchange between local features and global features can be carried out, resulting in a closer integration of the two, thus enhancing the comprehension of multi-scale information.

The core advantage of bidirectional cross attention is that it can dynamically extract information related to another feature from the perspective of global and local features at the same time, realizing deep-level information interaction. In traditional attention mechanisms, local and global features are often fused by simple weighting or splicing, which can easily lead to loss or redundancy of information. The bidirectional cross-attention mechanism, on the other hand, enables two features to guide each other, fully absorbing the important information of the other feature while retaining its own information, thus enhancing the expressive ability of the feature. For example, in computer vision tasks, this mechanism can help the model analyze the global structure while retaining key local details, making the final representation more complete and robust. Its core mathematical expression is as follows:

26

27

28

29

where represents the global feature map, extracted by a model component designed to capture long-range dependencies; represents the local feature map, extracted by a model component focused on local patterns; is the Global Average Pooling operation; is the Global Max Pooling operation; are distinct classification heads.

Feature fusion

We concatenate the extracted color, texture, and brightness features to form structured data. We can regard the image data as one modality and the hand-crafted features as another for multi-modal fusion⁵². Our proposed multi-modal fusion method is as follows: First, we use a multi-layer perceptron to learn the embedding representation of the hand-crafted data, and then concatenate it with the global features. After that, we use a linear layer to compress it back to the original dimension. The improved model structure diagram is shown in Fig. 7.

Fig. 7.

The architecture of enhanced HFANet model.

The core motivation for fusing hand-crafted global features with the global feature branch stems from the synergistic effects of three aspects: The hand-crafted color, texture, and other features, as explicit image-level statistics, have a natural consistency in semantic granularity with the global context features extracted by the global branch. They can be directly aligned and complementary at the macro level. Although the global branch can model long range dependencies, it has limitations in representing low level visual attributes (such as illumination robust color distribution). By fusing hand-crafted global features, we can inject physical prior knowledge at the same abstraction level, enhancing the model’s ability to perceive essential features. More importantly, as carriers of high-level semantics, global features need to fuse multi-modal information at an early stage to guide the subsequent interaction process. If fused with the local feature branch, the global statistical characteristics will be diluted by local spatial details. If the fusion is delayed to the classification head, it cannot fully influence the cross-modal interaction in the bidirectional cross-attention mechanism. Therefore, choosing to fuse at the global feature branch can ensure the efficient synergy of multi-modal information flow, so that the fused global features can directly participate in the query guidance in the subsequent attention calculation and avoid semantic deviations caused by cross-granularity fusion.

Experiments and results

Experiment setup

In order to verify the effectiveness of our proposed model for rock image classification, we compared the model performance on the sedimentary rock dataset with the following benchmark models: pre-trained VGG19 model, pre-trained ResNet34 model, pre-trained DenseNet121 model, pre-trained EfficientNet model, pre-trained vision transformer model, pre-trained swim transformer model, pre-trained ConvNeXt model⁵³, Rocks2Net model⁵, Enhanced_EfficientNet model³³, TLCA_ResNet model³⁴ and SENet_ConvNeXt model³⁵.

The dataset was divided into training, validation, and test sets in a 70:15:15 ratio while maintaining distributional consistency with the original data. To enhance model generalization, distinct preprocessing protocols were implemented: training data underwent augmentation through sequential resizing to 256 × 256, center cropping to 224 × 224, random horizontal flipping, and 90° rotation, followed by tensor conversion and normalization using ImageNet-standard mean and standard

deviation values of [0.485,0.456,0.406] and [0.229,0.224,0.225] respectively. Validation and test sets received identical resizing, cropping, and normalization without augmentation to ensure evaluation consistency. Early stopping terminated training upon five consecutive epochs without validation performance improvement, balancing robust feature learning with overfitting prevention through standardized preprocessing frameworks.

As detailed in the Data Description section and illustrated in Fig. 2, the dataset exhibits a notable class imbalance at the subtype level. To mitigate this issue during training, we apply a median-based undersampling approach to balance the class distribution. Specifically, for each data subset (igneous, metamorphic, and sedimentary), we first calculate the median sample count across all subtypes within that category. We then downsample all classes exceeding this median to twice the median value. For example, if the median sample count is 50, any class with more than 100 samples is randomly undersampled to 100 samples during training, while classes with fewer samples remain unchanged. This strategy effectively reduces the gap between majority and minority classes without being overly influenced by extreme outliers, while retaining sufficient samples for robust model learning. The undersampling is performed through random selection from the majority classes to ensure unbiased sample representation.

Furthermore, to ensure a robust and fair evaluation, we have chosen a suite of metrics that account for potential remaining imbalance. Alongside overall accuracy, we prioritize macro-averaged precision, macro-averaged recall, and the macro F1-score. These metrics give equal weight to each class by calculating the metric independently for each class and then taking the unweighted average, thus preventing the model’s performance on dominant classes from skewing the overall assessment. Additionally, we report the Area Under the ROC Curve (AUC) and the Area Under the Precision-Recall Curve (AUPR), which are also effective for evaluating performance on imbalanced datasets.

To comprehensively assess the effectiveness of the proposed model, we conducted a series of experiments covering different classification tasks and evaluation aspects. These include fine-grained rock subtype classification on each of the three sub-datasets (sedimentary, igneous, and metamorphic), and a three-class classification task on the merged dataset combining all subtypes. Additionally, we performed feature fusion experiments to examine the benefits of integrating handcrafted geological features with deep representations. Ablation studies were conducted to evaluate the contribution of key model components, followed by interpretability analysis to visualize and understand the model’s decision-making process.

To further evaluate the robustness and real-world applicability of our proposed model, we conducted cross-dataset generalization experiments using an independent mineral thin section dataset. This generalization test employed a different classification strategy based on established petrological principles where specific minerals serve as characteristic indicators of particular rock types: igneous indicators (olivine and augite), metamorphic indicators (andalusite, staurolite and actinolite), and sedimentary indicators (oolites). The generalization dataset was preprocessed using identical protocols as the primary dataset (resizing to 224 × 224, normalization with ImageNet statistics) but without any data augmentation, serving purely as an independent test set. Models trained on the primary dataset were directly applied to classify these mineral images into the corresponding three major rock categories, without any fine-tuning or adaptation. This experimental design evaluates the model’s ability to generalize beyond the training distribution when encountering geological samples with different imaging conditions, mineral compositions, and acquisition protocols.

To rigorously assess the statistical significance of the performance improvements achieved by our proposed model, we employed a bootstrap confidence interval approach for all performance metrics^54,55. Since both models were evaluated on the identical test set, their performance scores are inherently paired, necessitating statistical tests designed for dependent samples. We used a unified bootstrap method for Accuracy, F1-score, Precision, Recall, AUC, and AUPR. This non-parametric technique does not rely on distributional assumptions, which are often violated by machine learning performance metrics. The procedure involved generating 2000 bootstrap samples by resampling the original test set with replacement, calculating the performance difference for each sample to construct an empirical distribution. From this distribution, we constructed 95% confidence intervals, with statistical significance determined when the confidence interval did not contain zero.

Beyond significance testing, we computed effect sizes using Cohen’s d⁵⁶ to quantify the practical magnitude of differences. We employed a significance marking system: *** for large effects (|d| ≥ 0.8), ** for medium effects (0.5 ≤ |d| < 0.8), * for small effects (0.2 ≤ |d| < 0.5), † for trivial but significant effects, and “n.s.” for non-significant differences. Statistical significance was determined at level. This approach ensures that our claims of model superiority represent both statistically reliable and practically meaningful improvements.

Sedimentary rock subtype classification experiment

We evaluate the proposed model on the sedimentary rock sub-dataset, which involves a 12-class classification problem. The goal is to distinguish between different sedimentary rock subtypes, including hematite, shale, phosphorite, dolomite, siltstone, sandstone, siliceous rock, bauxite, tuff, subaqueous tuff, anhydrite and limestone. This experiment aims to assess the model’s capability in fine-grained classification and compare its performance against several established baseline models.

The experimental results of model performance comparison are summarized in Table 1. In the sedimentary rock image classification task, the HANet model outperforms all baseline models across multiple evaluation metrics. Specifically, HANet achieves the highest accuracy (0.9806), macro precision (0.9906), macro recall (0.9936), and macro F1-score (0.9918), indicating not only a high classification accuracy but also a balanced performance across categories. In addition, HANet also achieves the highest values in AUC (1.0000) and AUPR (1.0000), suggesting excellent discriminative power between positive and negative samples.

Table 1.

Performance comparison of different image classification models on sedimentary rock dataset.

Model	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
VGG19	0.9029	0.9581	0.8675	0.8904	0.9923	0.9779
ResNet34	0.9223	0.9361	0.9124	0.9138	0.9966	0.9920
DenseNet121	0.9320	0.9464	0.9494	0.9436	0.9988	0.9912
EfficientNet	0.8447	0.8840	0.8817	0.8678	0.9849	0.9166
Vison transformer	0.9320	0.9337	0.9312	0.9286	0.9959	0.9740
Swin transformer	0.9417	0.9748	0.9432	0.9554	0.9991	0.9964
ConvNeXt	0.9709	0.9694	0.9528	0.9596	0.9992	0.9936
Rocks2Net	0.9223	0.9048	0.8922	0.8878	0.9951	0.9694
Enhanced_ EfficientNet	0.9495	0.9330	0.9298	0.922	0.9993	0.9919
TLCA_ResNet	0.9596	0.9720	0.9474	0.9512	0.9986	0.9870
SENet_ConvNeXt	0.9293	0.9180	0.8722	0.8767	0.9971	0.9729
HFANet	0.9806	0.9906	0.9936	0.9918	1.0000	1.0000

The Bold value indicates best value of metrics

To rigorously validate the superiority of HFANet, we performed statistical significance testing using a bootstrap confidence interval approach with effect size analysis. The results of our comprehensive statistical analysis, comparing HFANet against the ConvNeXt baseline model, are summarized in Table 2. The bootstrap analysis with 2,000 iterations reveals that HFANet achieves statistically significant superiority in Macro AUC (95% CI: [-0.0712, + 0.1839], Cohen’s d = + 4.439) and Macro AUPR (95% CI: [+ 0.0192, + 0.0694], Cohen’s d = + 3.255), both demonstrating large effect sizes and substantial improvements in the model’s ranking and discrimination capabilities.

Table 2.

Performance metrics and statistical validation of HFANet against ConvNeXt baseline on sedimentary rock dataset.

Metrics	Significance	Cohen’s d	Improvement %	95% CI
Accuracy	n.s.	+ 0.997	+ 1.03%	[0.0000, + 0.0303]
Macro Precsion	n.s.	+ 0.489	+ 0.50%	[-0.0005, + 0.0920]
Macro Recall	n.s.	+ 0.999	+ 2.92%	[0.0000, + 0.0833]
Macro F1	n.s.	+ 0.894	+ 1.99%	[-0.0002, + 0.0879]
Macro AUC	***	+ 4.439	+ 23.68%	[+ 0.0712, + 0.1839]
Macro AUPR	***	+ 3.255	+ 14.19%	[+ 0.0192, + 0.0694]

While HFANet shows notable improvements in traditional classification metrics with large effect sizes (Cohen’s d > 0.8), these advantages did not achieve statistical significance at the α = 0.05 level due to higher variability in bootstrap sampling, as evidenced by confidence intervals containing zero. However, the statistically significant gains in Macro AUC and Macro AUPR—threshold-independent metrics that comprehensively evaluate ranking quality across all classification thresholds—provide robust evidence of HFANet’s superior discriminative capabilities for sedimentary rock classification.

To further investigate model performance at the class level, Fig. 8 presents the F1-scores of each model across the 12 sedimentary rock categories. As shown in the figure, HANet demonstrates consistently high performance across all categories, achieving perfect or near-perfect F1-scores in each case. In contrast, other models such as VGG19, ResNet34, and even transformer-based approaches like Swin Transformer exhibit larger fluctuations, particularly on categories such as shale. This result confirms that HANet is more robust and reliable in handling the subtle visual differences among rock subtypes.

Fig. 8.

Class-wise F1-score comparison of different models on the sedimentary rock sub-dataset.

Overall, the experimental results highlight that HANet achieves optimal performance in the sedimentary rock classification task and shows strong potential for practical geological applications. Its superior performance over both traditional CNNs and transformer-based models reflects its enhanced feature extraction capability, particularly for complex and texture-rich rock imagery. The model not only surpasses Swin Transformer in terms of precision and recall, but also achieves perfect AUC and AUPR values, confirming its robustness in handling multi-class scenarios and reducing misclassification errors.

Igneous rock subtype classification experiment

In this experiment, we evaluate the performance of HFANet on the igneous rock sub-dataset, which involves a six-class classification task. The goal is to distinguish between six representative igneous rock subtypes, namely basaltic rock, alkaline rock with related rocks, intermediate rock, ultrabasic (ultramafic) rock, acidic rock and ultrabasic rock. This task aims to assess the model’s capability in fine-grained classification across diverse mineral compositions and structural variations in igneous rock imagery. Table 3 summarizes the performance metrics of each model. HFANet consistently outperforms all baselines, demonstrating superior overall performance. As shown in Table 2, HFANet achieves the highest scores across all metrics, with an accuracy of 91.49%, macro F1-score of 92.90%, and AUC and AUPR reaching 0.9946 and 0.9820, respectively, indicating strong discriminative power and robustness.

Table 3.

Performance comparison of different image classification models on igneous rock dataset.

Model	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
VGG19	0.7589	0.8034	0.8044	0.7982	0.9508	0.8587
ResNet34	0.7518	0.7923	0.7252	0.7463	0.9495	0.8447
DenseNet121	0.7589	0.7987	0.8030	0.7943	0.9728	0.9178
EfficientNet	0.8652	0.8960	0.8717	0.8792	0.9754	0.9431
Vison transformer	0.8936	0.9248	0.9043	0.9087	0.9913	0.9700
Swin transformer	0.8794	0.8640	0.8981	0.8765	0.9886	0.9641
ConvNeXt	0.8865	0.9084	0.9181	0.9085	0.9901	0.9677
Rocks2Net	0.6525	0.7495	0.6685	0.6800	0.9345	0.8106
Enhanced_EfficientNet	0.7660	0.8169	0.7962	0.8017	0.9661	0.8849
TLCA_ResNet	0.8014	0.8106	0.8215	0.7887	0.9716	0.9254
SENet_ConvNeXt	0.8156	0.7975	0.8474	0.8105	0.9679	0.8891
HFANet	0.9149	0.9342	0.9275	0.9290	0.9946	0.9820

The Bold value indicates best value of metrics

To rigorously validate the superiority of HFANet, we performed statistical significance testing using a bootstrap confidence interval approach with effect size analysis. The results of our comprehensive statistical analysis, comparing HFANet against the Vision Transformer baseline model, are summarized in Table 4. The bootstrap analysis reveals that HFANet achieves statistically significant superiority across the majority of key evaluation metrics, with large effect sizes demonstrated in Accuracy (Cohen’s d = + 2.885), Macro Precision (Cohen’s d = + 2.049), Macro Recall (Cohen’s d = + 2.323), Macro F1 (Cohen’s d = + 2.263), and Macro AUPR (Cohen’s d = + 2.226), all showing substantial improvements ranging from 3.87% to 12.06%.

Table 4.

Performance metrics and statistical validation of HFANet against Vison transformer baseline on IGNEOUS rock dataset.

Metrics	Significance	Cohen’s d	Improvement %	95% CI
Accuracy	***	+ 2.885	+ 10.00%	[+ 0.0284, + 0.1418]
Macro Precsion	***	+ 2.049	+ 11.83%	[+ 0.0180, + 0.2095]
Macro Recall	***	+ 2.323	+ 10.39%	[+ 0.0342, + 0.2181]
Macro F1	***	+ 2.263	+ 12.06%	[+ 0.0297, + 0.2264]
Macro AUC	n.s.	+ 1.567	+ 0.90%	[-0.0018, + 0.0204]
Macro AUPR	***	+ 2.226	+ 3.87%	[+ 0.0066, + 0.0720]

While Macro AUC shows a notable effect size (Cohen’s d = + 1.567) with a + 0.90% improvement, it did not achieve statistical significance at the α = 0.05 level, as evidenced by its confidence interval [-0.0018, + 0.0204] containing zero. Nevertheless, the overwhelming and statistically significant gains across Accuracy, Precision, Recall, F1 Score, and AUPR robustly demonstrate that HFANet’s performance enhancements are genuine and represent substantial practical improvements. This comprehensive validation confirms HFANet’s superior ability to effectively classify igneous rocks, maintain balanced precision-recall performance, and accurately discriminate between different rock types.

To gain further insight into model behavior at the class level, Fig. 9 presents the F1-score for each model across the six igneous rock subtypes. HFANet demonstrates consistently strong performance across all categories, achieving near-perfect F1-scores in intermediate rocks, felsic rocks, and ultramafic rocks. In contrast, traditional convolution-based models such as VGG19 and ResNet34 exhibit larger performance fluctuations, particularly in categories such as alkaline rock with related rocks and inter-mediate rock, where F1-scores drop to 0.59 and 0.65, respectively. This highlights the challenge these models face when dealing with classes exhibiting high inter-class similarity and complex visual textures.

Fig. 9.

Class-wise F1-score comparison of different models on the igneous rock sub-dataset.

Overall, HFANet not only achieves the highest overall classification accuracy but also maintains high and stable performance across all igneous rock subtypes. These results validate the model’s robustness and generalization ability in handling fine-grained geological image classification tasks with diverse morphological patterns.

Metamorphic rock subtype classification experiment

To comprehensively assess model performance on structurally complex and mineralogically diverse rocks, we conducted experiments on the metamorphic rock sub-dataset. This dataset includes 17 distinct metamorphic rock subtypes: hornfels, migmatite, granulite, metamorphic rocks with relict structures, slate, skarn, phyllite, gneiss, schist, serpentinite, cataclasite, marble, granulitic gneiss, eclogite, lepidolite, mylonite, and plagioclase amphibolite. As summarized in Table 5, HFANet achieves the best results in most evaluation metrics among the compared models, including accuracy (0.9928), macro precision (0.9882), macro recall (0.9853), and macro F1-score (0.9851). While HFANet slightly trails Swin Transformer and ConvNeXt in terms of AUC (0.9983 vs. 0.9996 and 0.9990) and AUPR (0.9882 vs. 0.9915), its overall performance remains the most consistent and balanced across all categories.

Table 5.

Performance comparison of different image classification models on metamorphic rock dataset.

Model	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
VGG19	0.8201	0.8488	0.7963	0.7886	0.9876	0.9095
ResNet34	0.9281	0.9472	0.9249	0.9274	0.9985	0.9833
DenseNet121	0.9209	0.9510	0.9092	0.9230	0.9963	0.9587
EfficientNet	0.8993	0.8960	0.9156	0.8976	0.9891	0.9603
Vison transformer	0.9568	0.9692	0.9396	0.9421	0.9956	0.9595
Swin transformer	0.9712	0.9774	0.9568	0.9641	0.9996	0.9909
ConvNeXt	0.9712	0.9674	0.9470	0.9512	0.9990	0.9915
Rocks2Net	0.8058	0.8655	0.8346	0.8146	0.9948	0.9598
Enhanced_EfficientNet	0.8777	0.8830	0.8489	0.8445	0.9753	0.9175
TLCA_ResNet	0.8921	0.9023	0.8639	0.8631	0.9946	0.9512
SENet_ConvNeXt	0.9065	0.9090	0.8952	0.8856	0.9974	0.9667
HFANet	0.9928	0.9882	0.9853	0.9851	0.9983	0.9882

The Bold value indicates best value of metrics

To rigorously validate the superiority of HFANet, we performed statistical significance testing using a bootstrap confidence interval approach with effect size analysis. The results of our comprehensive statistical analysis, comparing HFANet against the Swin Transformer baseline model, are summarized in Table 6. The bootstrap analysis reveals that HFANet achieves statistically significant superiority in Accuracy (Cohen’s d = + 2.536, + 6.56% improvement, 95% CI: [+ 0.0144, + 0.1079]), demonstrating a large effect size and substantial improvement in overall classification performance on the metamorphic rock dataset.

Table 6.

Performance metrics and statistical validation of HFANet against Swin transformer baseline on METAMORPHIC rock dataset.

Metrics	Significance	Cohen’s d	Improvement %	95% CI
Accuracy	***	+ 2.536	+ 6.56%	[+ 0.0144, + 0.1079]
Macro Precsion	n.s.	+ 1.071	+ 2.55%	[− 0.0264, + 0.1245]
Macro Recall	n.s.	+ 1.701	+ 5.59%	[− 0.0075, + 0.1032]
Macro F1	n.s.	+ 1.478	+ 5.39%	[− 0.0137, + 0.1170]
Macro AUC	n.s.	− 1.172	− 4.35%	[− 0.0581, + 0.0161]
Macro AUPR	n.s.	+ 0.426	+ 5.32%	[− 0.0211, + 0.0442]

However, the remaining evaluation metrics did not achieve statistical significance at the α = 0.05 level, with confidence intervals containing zero for Macro Precision, Macro Recall, Macro F1, and Macro AUPR, despite showing moderate to large effect sizes in most cases. Notably, Macro AUC showed a slight decrease (− 4.35%, Cohen’s d = − 1.172), though this difference was not statistically significant. While the significant improvement in Accuracy demonstrates HFANet’s enhanced overall classification capability, the mixed results across other metrics suggest that the performance gains on metamorphic rock classification are more moderate compared to the substantial improvements observed on sedimentary and igneous rock datasets, indicating potential dataset-specific characteristics that may influence model performance.

Figure 10 presents the per-class F1-scores for each model across the 17 metamorphic rock subtypes. HFANet delivers remarkably stable and high performance across nearly all classes, achieving perfect (1.00) or near-perfect F1-scores in most categories. Particularly noteworthy is its robustness in classes such as eclogite, lepidolite, and migmatite, where many competing models exhibit significant performance drops. For instance, VGG19 struggles with lepidolite and metamorphic rocks with relict structures, where F1-scores fall to 0.40 and 0.63 respectively. Rocks2Net also shows considerable inconsistency across categories.

Fig. 10.

Class-wise F1-score comparison of different models on the metamorphic rock sub-dataset.

In conclusion, although HFANet does not achieve the highest AUC and AUPR, it consistently outperforms all baseline models in terms of classification accuracy and macro-level performance, highlighting its superior generalization and fine-grained recognition capabilities. Its dual-branch structure and hybrid attention mechanisms prove especially effective in managing the subtle visual cues and structural complexities of metamorphic rock imagery.

Three main rock type classification experiment

In this experiment, we thoroughly evaluated the performance of HFANet on a comprehensive rock dataset, which encompasses three primary rock types: sedimentary, igneous, and metamorphic rocks. The objective was to assess HFANet’s classification capabilities across these major rock types and compare its performance against several baseline models. As detailed in Table 4, HFANet demonstrates state-of-the-art performance, surpassing all other models. Specifically, with SENet_ConvNeXt now established as the second-best model, HFANet maintains a clear and consistent advantage. Our model achieves an accuracy of 0.9661, which is a notable improvement over SENet_ConvNeXt’s 0.9525. This superior performance extends across macro precision (0.9685 vs. 0.9518), macro recall (0.9657 vs. 0.9560), and the macro F1-score (0.9670 vs. 0.9526). Furthermore, its values for AUC (0.9979) and AUPR (0.9965) are exceptionally high, indicating near-perfect discriminative power.

To rigorously validate the superiority of HFANet, we performed statistical significance testing using a bootstrap confidence interval approach with effect size analysis. The results of our comprehensive statistical analysis, comparing HFANet against the SENET_ConvNeXt baseline model on the full rock dataset, are summarized in Table 7. The bootstrap analysis reveals that HFANet achieves statistically significant superiority across all evaluation metrics, with large effect sizes demonstrated in Accuracy (Cohen’s d = + 2.752), Macro Precision (Cohen’s d = + 2.874), Macro Recall (Cohen’s d = + 2.877), Macro F1 (Cohen’s d = + 2.756), Macro AUC (Cohen’s d = + 3.687), and Macro AUPR (Cohen’s d = + 3.781), showing consistent improvements ranging from 1.10% to 5.26%.

Table 7.

Performance metrics and statistical validation of HFANet against SENET_ConvNeXt baseline on full rock dataset.

Metrics	Significance	Cohen’s d	Improvement %	95% CI
Accuracy	***	+ 2.752	+ 5.26%	[+ 0.0158, + 0.0818]
Macro Precsion	***	+ 2.874	+ 4.99%	[+ 0.0157, + 0.0772]
Macro Recall	***	+ 2.877	+ 5.16%	[+ 0.0164, + 0.0796]
Macro F1	***	+ 2.756	+5.06%	[+ 0.0148, + 0.0795]
Macro AUC	***	+ 3.687	+ 1.10%	[+ 0.0056, + 0.0172]
Macro AUPR	***	+ 3.781	+ 1.87%	[+ 0.0097, + 0.0285]

The comprehensive statistical validation demonstrates HFANet’s robust and consistent superiority across all key performance dimensions on the complete rock classification task. All confidence intervals exclude zero, confirming that these improvements are statistically significant and not due to random variation. The particularly strong effect sizes in Macro AUC and Macro AUPR (Cohen’s d > 3.6) indicate substantial enhancements in the model’s ranking and discrimination capabilities across all rock types. This comprehensive validation provides compelling evidence that HFANet’s architectural innovations translate into genuine and substantial performance gains for multi-class geological rock classification across diverse rock categories.

Figure 11 presents the per-class F1-scores for each model across the three main rock types: sedimentary, igneous, and metamorphic rocks. HFANet demonstrates exceptional performance in terms of F1 scores across the three major rock types: sedimentary, igneous, and metamorphic rocks. HFANet achieves F1 scores of 0.98, 0.96, and 0.97 for sedimentary, igneous, and metamorphic rocks, respectively, indicating its high stability and accuracy in classification. In contrast, traditional models such as VGG19 exhibit notable deficiencies. VGG19 attains F1 scores of 0.58, 0.72, and 0.68, respectively, with particularly poor performance in classifying sedimentary rocks. Although ConvNeXt, a stronger comparative model, achieves F1 scores of 0.95, 0.89, and 0.91, it still falls short of HFANet in certain categories. This highlights HFANet’s distinct advantages in rock classification, especially its robustness and accuracy in handling complex rock type.

Fig. 11.

Class-wise F1-score comparison of different models on the full rock sub-dataset.

In conclusion, HFANet demonstrates superior performance in classifying the three primary rock types—sedimentary, igneous, and metamorphic rocks—by consistently outperforming the baseline models in terms of accuracy, macro-level metrics, and per-class F1 scores. While ConvNeXt and other advanced models show strong performance in certain aspects, HFANet’s near-perfect AUC and AUPR values, combined with its high F1 scores across all rock types, highlight its exceptional generalization and robustness. This comprehensive evaluation underscores HFANet’s potential as a leading solution for rock classification tasks.

Cross-dataset generalization experiments

To further evaluate the robustness and real-world applicability of HFANet, we conducted cross-dataset generalization experiments using an independent mineral thin section dataset. This generalization test represents a challenging evaluation scenario where models trained on the primary rock dataset are directly applied to classify mineral images from a different acquisition protocol and imaging conditions without any fine-tuning or adaptation. The results reveal a dramatic and counterintuitive finding: VGG19, which performed worst on the original training dataset (Table 8: accuracy 0.6745, ranking last among all models), demonstrates the strongest generalization performance, while HFANet, despite achieving near-perfect performance on the original dataset, ranks seventh in generalization accuracy, highlighting a fundamental trade-off between in-domain optimization and cross-domain transferability in geological deep learning applications.

Table 8.

Performance comparison of different image classification models on full rock dataset.

Model	Accuracy	Macro precision	Macro recall	Macro F1 Score	AUC	AUPR
VGG19	0.6745	0.7293	0.6535	0.6603	0.8873	0.8137
ResNet34	0.7500	0.7644	0.7563	0.7454	0.9235	0.8656
DenseNet121	0.8333	0.8351	0.8342	0.8335	0.9597	0.9313
EfficientNet	0.8099	0.8260	0.8034	0.8096	0.9389	0.8878
Vison transformer	0.8909	0.8935	0.8968	0.8930	0.9810	0.9672
Swin transformer	0.8568	0.8614	0.8598	0.8587	0.9736	0.9543
ConvNeXt	0.9141	0.9158	0.9191	0.9173	0.9845	0.9726
Rocks2Net	0.5391	0.7488	0.4976	0.4518	0.8155	0.7147
Enhanced_ EfficientNet	0.8311	0.8641	0.8225	0.8319	0.9565	0.9274
TLCA_ResNet	0.8918	0.8924	0.9025	0.8912	0.9863	0.9741
SENet_ConvNeXt	0.9525	0.9518	0.956	0.9526	0.9944	0.9893
HFANet	0.9661	0.9685	0.9657	0.9670	0.9979	0.9965

The Bold value indicates best value of metrics

As detailed in Table 9, the generalization results show a striking performance reversal compared to the original dataset rankings. VGG19 achieves the highest accuracy (0.6901), followed by Rocks2Net (0.6746) and ConvNeXt (0.6648), while HFANet ranks seventh with an accuracy of 0.6028. This represents a dramatic fall from HFANet’s top-tier original performance (0.9661) to a mid-tier generalization result. However, an interesting pattern emerges when examining the discriminative metrics: HFANet achieves the highest AUC (0.8905) and AUPR (0.7881) values in the generalization test, outperforming even VGG19 (AUC: 0.8672, AUPR: 0.7436). This suggests that while HFANet struggles with classification accuracy across different domains, its learned feature representations maintain superior ranking and discrimination capabilities. The disparity between classification metrics (accuracy, precision, recall, F1) and discriminative metrics (AUC, AUPR) indicates that HFANet’s sophisticated attention mechanisms preserve relative confidence ordering between classes even when absolute classification decisions become less reliable in cross-domain scenarios.

Table 9.

Generalization comparison of different image classification models on mineral thin section dataset.

Model	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
VGG19	0.6901	0.6634	0.6436	0.6360	0.8672	0.7436
ResNet34	0.5732	0.5194	0.5576	0.475	0.7939	0.6077
DenseNet121	0.6380	0.6569	0.6007	0.5519	0.8633	0.7746
EfficientNet	0.4592	0.4556	0.5164	0.4542	0.6773	0.5292
Vison transformer	0.5972	0.4918	0.4682	0.4611	0.7918	0.5681
Swin transformer	0.5944	0.5985	0.5205	0.461	0.7995	0.6204
ConvNeXt	0.6648	0.7657	0.6311	0.5985	0.8173	0.7342
Rocks2Net	0.6746	0.7302	0.5742	0.5975	0.6962	0.5558
Enhanced_EfficientNet	0.5127	0.5033	0.5568	0.5021	0.7124	0.5682
TLCA_ResNet	0.6099	0.6128	0.5319	0.5176	0.7244	0.5624
SENet_ConvNeXt	0.6239	0.5789	0.5624	0.5544	0.8099	0.6682
HFANet	0.6028	0.7118	0.5276	0.4896	0.8905	0.7881

The Bold value indicates best value of metrics

The statistical significance analysis presented in Table 10 provides rigorous validation of the generalization performance comparison between HFANet and the best-performing VGG19 baseline. The bootstrap analysis reveals significant performance gaps: HFANet shows statistically significant decreases in Accuracy (Cohen’s d = − 4.636, − 12.65%), Macro Recall (Cohen’s d = − 4.915, − 18.02%), and Macro F1 (Cohen’s d = − 5.849, − 23.03%) compared to VGG19. However, HFANet demonstrates statistically significant improvement in Macro AUPR (Cohen’s d = + 1.952, + 6.00%), with no significant difference in Macro Precision and Macro AUC, suggesting that while overall classification performance suffers, the model’s discriminative capabilities remain robust. These mixed results highlight the complex nature of cross-domain generalization in geological applications, where sophisticated architectural innovations may excel at feature discrimination but struggle with definitive classification decisions when encountering new imaging protocols and mineral compositions.

Table 10.

Performance metrics and statistical validation of HFANet against VGG19 baseline on mineral thin section dataset.

Metrics	Significance	Cohen’s d	Improvement %	95% CI
Accuracy	***	− 4.636	− 12.65%	[− 0.1239, − 0.0521]
Macro Precsion	n.s.	+ 1.256	+ 7.29%	[− 0.0291, + 0.1214]
Macro Recall	***	− 4.915	− 18.02%	[− 0.1620, − 0.0698]
Macro F1	***	− 5.849	− 23.03%	[− 0.1948, − 0.0987]
Macro AUC	n.s.	+ 1.789	+ 2.69%	[− 0.0016, + 0.0496]
Macro AUPR	***	+ 1.952	+ 6.00%	[+ 0.0005, + 0.0913]

This apparent paradox reflects a fundamental trade-off between domain-specific optimization and cross-domain generalization. VGG19, ranking last on the original dataset (Table 8: 67.45%), achieves the highest cross-domain accuracy (Table 9: 69.01%) precisely because its simple architecture fails to capture domain-specific knowledge of rock thin sections, learning only generic transferable features. In contrast, HFANet’s near-perfect in-domain performance (96.61%) demonstrates deep learning of unique petrographic characteristics(subtle textures, mineralogical relationships, and structural patterns)through its dual-branch attention mechanism. This specialization causes substantial accuracy degradation (37.6%) when transferred to the mineral dataset with different imaging protocols. However, HFANet achieves the highest cross-domain AUC (0.8905) and AUPR (0.7881), substantially outperforming VGG19 (AUC: 0.8672, AUPR: 0.7436). This indicates HFANet’s learned features maintain superior discriminative power across domains despite unreliable decision boundaries. The disparity between accuracy degradation (37.6%) and AUC degradation (10.8%) reveals that the dual-branch architecture successfully captures transferable discriminative representations but becomes sensitive to domain-specific classification thresholds. This suggests domain adaptation techniques focusing on decision boundary calibration, such as adversarial domain alignment or confidence adjustment, would effectively improve HFANet’s cross-domain accuracy while leveraging its already robust discriminative features.

To provide a more comprehensive evaluation of cross-dataset generalization beyond classification accuracy, Fig. 12 presents the discriminative performance comparison through class-wise AUC and AUPR metrics. These ranking-based metrics reveal a strikingly different pattern from the accuracy-based results presented in Table 9, demonstrating that HFANet maintains superior discriminative capabilities despite lower classification accuracy in cross-domain scenarios.

Fig. 12.

Discriminative performance comparison on mineral thin section dataset (generalization test). a Class-wise AUC comparison across 12 models. b Class-wise AUPR comparison across 12 models.

Sedimentary rocks show the largest performance variance and HFANet’s strongest advantage, achieving exceptional scores (AUC: 0.9682, AUPR: 0.8000) that substantially outperform VGG19 (AUC: 0.8766, AUPR: 0.6037) by + 9.16 and + 19.63% points respectively. This suggests HFANet’s attention mechanisms effectively capture transferable sedimentary textures such as grain boundaries and matrix structures. In contrast, Igneous rocks exhibit the most consistent cross-model performance, with AUC scores clustering tightly between 0.76 and 0.89. This indicates crystalline features are inherently robust to imaging variations. HFANet maintains competitive performance (AUC: 0.8936, AUPR: 0.8733) with modest advantages, as the distinctive optical properties of igneous minerals provide strong discriminative signals accessible to multiple architectures. However, Metamorphic rocks present the greatest generalization challenge. HFANet (AUC: 0.8098, AUPR: 0.6911) trails VGG19 (AUC: 0.8439, AUPR: 0.7318), suggesting complex foliation patterns and recrystallization textures are more sensitive to domain shift. This heterogeneous pattern reveals HFANet exhibits a specialist profile, performing exceptionally on Sedimentary rocks, remaining competitive on Igneous rocks, but showing moderate performance on Metamorphic rocks. This indicates domain adaptation may be necessary for comprehensive cross-domain deployment.

An intriguing observation from Table 9 warrants deeper examination that VGG19, despite its architectural simplicity, achieves higher classification accuracy (0.6317) than HFANet (0.6028) in the cross-dataset generalization test. This apparent paradox can be understood through a comprehensive analysis of performance degradation patterns across domains. Comparing the in-domain performance (Table 8) with the cross-domain results (Table 9), VGG19 exhibits a modest accuracy decline of 6.3% (from 0.6745 to 0.6317), while HFANet experiences a substantially larger drop of 37.6% (from 0.9661 to 0.6028). However, this disparity does not necessarily indicate superior generalization capability of VGG19. Rather, it reflects a performance ceiling effect where VGG19’s mediocre in-domain performance (ranking last among all models in Table 8 with 67.45% accuracy) leaves limited room for further degradation under domain shift. In contrast, HFANet starts from near-perfect in-domain performance, resulting in a larger absolute decline despite maintaining competitive cross-domain accuracy.

Crucially, discriminative metrics reveal a more nuanced picture. As shown in Fig. 12, HFANet achieves the highest AUC (0.8905) and AUPR (0.7881) in cross-domain evaluation, surpassing VGG19’s AUC (0.8672) and AUPR (0.7436). The relative AUC degradation of HFANet (10.8%, from 0.9979 to 0.8905) is substantially lower than its accuracy degradation (37.6%), indicating that the learned feature representations maintain strong discriminative power across domains, while the decision boundaries become less reliable. This suggests that HFANet’s sophisticated attention mechanisms successfully capture transferable discriminative features but are sensitive to domain-specific variations in classification thresholds. Conversely, VGG19’s simpler convolutional architecture provides generic inductive biases that yield stable but less discriminative decision boundaries.

This performance pattern highlights the distinction between classification accuracy and discriminative capability under domain shift. HFANet’s superior ranking quality (AUC/AUPR) despite lower accuracy indicates that its dual-branch attention architecture learns more powerful feature representations that preserve relative confidence ordering across domains, even when absolute classification decisions require domain-specific calibration. This finding suggests that domain adaptation techniques focusing on decision boundary adjustment rather than feature re-learning, such as adversarial domain adaptation or confidence calibration, would be particularly effective for improving HFANet’s cross-domain classification accuracy while leveraging its already superior discriminative features. Future work will explore these domain adaptation strategies to bridge the gap between in-domain and cross-domain performance while maintaining the architectural advantages that drive HFANet’s exceptional discriminative power.

Ablation experiment

To comprehensively evaluate the contributions of individual components within HFANet, we conducted detailed ablation studies on multiple datasets, including the sedimentary rock subtype dataset, the igneous rock dataset, the metamorphic rock dataset, and the full rock dataset. The experiments were designed to progressively disable key modules and adjust the network structure to isolate the impact of each component.

Sedimentary rock subset

The results of the ablation experiment on the sedimentary rock subtype dataset are summarized in Table 11. The global branch, using only the Swin Transformer for global feature extraction, achieved an accuracy of 91.26%, with macro precision, recall, and F1-score of 92.31%, 93.74%, and 92.47%, respectively. This indicates that the global branch alone can capture significant contextual information but is insufficient for fine-grained classification.

Table 11.

Result of the ablation experiment on sedimentary rock dataset.

Component	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
Global Branch	0.9126	0.9231	0.9374	0.9247	0.9982	0.9920
Local Branch1	0.9126	0.9308	0.9016	0.9126	0.9870	0.9653
Local Branch2	0.9417	0.9760	0.9215	0.9422	0.9948	0.9750
EFANet-	0.9515	0.9520	0.9640	0.9541	0.9997	0.9988
EFANet-	0.9612	0.9521	0.9640	0.9539	0.9995	0.9975
HFANet	0.9806	0.9906	0.9936	0.9918	1.0000	1.0000

The Bold value indicates best value of metrics

The local branch variants, which rely solely on DenseNet for local feature extraction, demonstrated varying performance. The local branch with max pooling (Local Branch2) achieved higher accuracy (94.17%) and macro F1-score (94.22%) compared to the average pooling variant (Local Branch1), suggesting that max pooling is more effective in preserving salient local texture features critical for rock classification.

The simplified dual-branch networks without attention mechanisms showed substantial improvements. The HFANet– variant, which includes neither the multi-head self-attention mechanism nor the bidirectional cross-attention module, achieved an accuracy of 95.15% and a macro F1-score of 95.41%. Adding the multi-head self-attention mechanism (HFANet-) further improved performance to 96.12% accuracy and 95.39% macro F1-score. The full HFANet model, incorporating both attention mechanisms and the ensemble classification framework, achieved the highest accuracy (98.06%), macro F1-score (99.18%), and perfect AUC and AUPR values (1.0000). This demonstrates that both attention mechanisms and the ensemble classification strategy significantly enhance the model’s performance on fine-grained sedimentary rock classification.

Igneous rock dataset

The ablation results on the igneous rock dataset are presented in Table 12. The global branch achieved an accuracy of 87.94%, with macro precision, recall, and F1-score of 89.53%, 89.55%, and 88.70%, respectively. This indicates that the global branch can capture significant contextual information but is less effective for fine-grained classification compared to the sedimentary rock dataset.

Table 12.

Result of the ablation experiment on igneous rock dataset.

Component	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
Global Branch	0.8794	0.8953	0.8955	0.8870	0.9945	0.9786
Local Branch1	0.6950	0.8238	0.6739	0.6899	0.9600	0.8374
Local Branch2	0.8511	0.8730	0.8766	0.8691	0.9848	0.9550
EFANet-	0.9149	0.9394	0.8714	0.8976	0.9877	0.9374
EFANet-	0.9149	0.8959	0.9288	0.9072	0.9890	0.9542
HFANet	0.9149	0.9342	0.9275	0.9290	0.9946	0.9820

The Bold value indicates best value of metrics

The local branch with max pooling (Local Branch2) achieved an accuracy of 85.11% and a macro F1-score of 86.91%, while the average pooling variant (Local Branch1) performed poorly with an accuracy of 69.50% and a macro F1-score of 68.99%. This highlights the importance of max pooling in preserving salient local features for igneous rock classification.

The HFANet– variant achieved an accuracy of 91.49% and a macro F1-score of 89.76%, while the HFANet- variant further improved to 91.49% accuracy and 90.72% macro F1-score. The full HFANet model achieved the highest accuracy (91.49%), macro precision (93.42%), and macro recall (92.75%), with AUC and AUPR values of 0.9946 and 0.9820, respectively. This demonstrates that the attention mechanisms and ensemble classification strategy significantly enhance the model’s performance on igneous rock classification.

Metamorphic rock dataset

The ablation results on the metamorphic rock dataset are shown in Table 13. The global branch achieved an accuracy of 97.12%, with macro precision, recall, and F1-score of 96.74%, 95.46%, and 95.69%, respectively. This indicates strong performance in capturing global contextual information.

Table 13.

Result of the ablation experiment on metamorphic rock dataset.

Component	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
Global branch	0.9712	0.9674	0.9546	0.9569	0.9988	0.9795
Local branch1	0.8561	0.9539	0.8543	0.8831	0.9961	0.9804
Local branch2	0.9568	0.9434	0.9472	0.9372	0.9924	0.9685
EFANet-	0.9712	0.9711	0.9725	0.9692	0.9992	0.9942
EFANet-	0.9712	0.9809	0.9719	0.9742	0.9995	0.9958
HFANet	0.9928	0.9882	0.9853	0.9851	0.9983	0.9882

The Bold value indicates best value of metrics

The local branch with max pooling (Local Branch2) achieved an accuracy of 95.68% and a macro F1-score of 93.72%, while the average pooling variant (Local Branch1) performed slightly worse with an accuracy of 85.61% and a macro F1-score of 88.31%. The HFANet– variant achieved an accuracy of 97.12% and a macro F1-score of 96.92%, while the HFANet- variant further improved to 97.12% accuracy and 97.42% macro F1-score. The full HFANet model achieved the highest accuracy (99.28%), macro precision (98.82%), and macro recall (98.53%), with AUC and AUPR values of 0.9983 and 0.9882, respectively. This demonstrates that the attention mechanisms and ensemble classification strategy significantly enhance the model’s performance on metamorphic rock classification.

Full rock dataset

The ablation results on the full rock dataset, encompassing all three primary rock types, are summarized in Table 14. The global branch achieved an accuracy of 86.20%, with macro precision, recall, and F1-score of 86.08%, 86.74%, and 86.20%, respectively. This indicates that the global branch alone can capture significant contextual information but is insufficient for comprehensive rock classification.

Table 14.

Result of the ablation experiment on full rock dataset.

Component	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
Global branch	0.8620	0.8608	0.8674	0.8620	0.9702	0.9505
Local branch1	0.8385	0.8581	0.8510	0.8367	0.9824	0.9715
Local branch2	0.9167	0.9208	0.9230	0.9183	0.9902	0.9833
EFANet--	0.9453	0.9447	0.9450	0.9446	0.9915	0.9845
EFANet-	0.9479	0.9473	0.9526	0.9490	0.9953	0.9916
HFANet	0.9661	0.9685	0.9657	0.9670	0.9979	0.9965

The Bold value indicates best value of metrics

The local branch with max pooling (Local Branch2) achieved an accuracy of 91.67% and a macro F1-score of 91.83%, while the average pooling variant (Local Branch1) performed slightly worse with an accuracy of 83.85% and a macro F1-score of 83.67%. The HFANet– variant achieved an accuracy of 94.53% and a macro F1-score of 94.46%, while the HFANet- variant further improved to 94.79% accuracy and 94.90% macro F1-score. The full HFANet model achieved the highest accuracy (96.61%), macro precision (96.85%), and macro recall (96.57%), with AUC and AUPR values of 0.9979 and 0.9965, respectively. This demonstrates that the attention mechanisms and ensemble classification strategy significantly enhance the model’s performance on the full rock dataset.

Results and analysis

The comprehensive ablation studies across multiple datasets demonstrate the significant contributions of each component in HFANet. The global branch captures essential contextual information, while the local branch with max pooling effectively preserves salient local features. The multi-head self-attention mechanism and bidirectional cross-attention module significantly enhance the model’s ability to focus on critical regions and integrate multi-scale features. The ensemble classification framework further improves robustness and discriminative power. The full HFANet model consistently outperforms all simplified variants, highlighting the importance of each component in achieving optimal performance for fine-grained rock image classification.

Feature fusion verification experiment

To further evaluate the effectiveness of combining handcrafted geological features with deep image representations, we conducted feature fusion experiments on both the sedimentary rock sub-dataset and the igneous rock dataset. These experiments compared three types of input configurations: handcrafted features only, image features only, and a fusion of both. The goal was to assess whether the integration of hand-crafted features could enhance the classification performance of deep learning models.

Sedimentary rock subtype dataset

The results of the feature fusion experiment on the sedimentary rock sub-dataset are summarized in Table 15. The model trained solely on handcrafted features achieved limited performance, with an accuracy of 67.96%, macro precision of 70.17%, macro recall of 68.43%, and a macro F1-score of 64.47%. This indicates that handcrafted features alone are insufficient for fine-grained classification due to their limited ability to capture complex patterns in rock images.

Table 15.

Performance comparison of different feature classification models on sedimentary rock dataset.

Feature	Accuracy	Macro precision	Macro recall	Macro F1 score	AUC	AUPR
Image feature	0.9806	0.9906	0.9936	0.9918	1.0000	1.0000
Manual feature	0.6796	0.7017	0.6843	0.6447	0.9106	0.7318
All features	0.Feaut9903	0.9969	0.9944	0.9956	1.0000	1.0000

The Bold value indicates best value of metrics

In contrast, using image features alone resulted in significantly better outcomes, achieving 98.06% accuracy and perfect AUC and AUPR scores (1.0000). This demonstrates the powerful representational capacity of deep visual models on sedimentary rock imagery.

When combining handcrafted and image features using our proposed multimodal fusion strategy, all metrics further improved: accuracy reached 99.03%, macro precision rose to 99.69%, and the macro F1-score increased to 99.56%. This confirms that handcrafted features can provide complementary information to deep features, particularly in enhancing low-level color and texture cues. The fusion of handcrafted and deep features thus enhances the model’s ability to distinguish between subtle visual differences in sedimentary rock subtypes.

Igneous rock dataset

The results of the feature fusion experiment on the igneous rock dataset are presented in Table 16. Similar to the sedimentary rock dataset, the model trained solely on handcrafted features achieved poor performance, with an accuracy of only 25.53%, macro precision of 9.08%, macro recall of 17.09%, and a macro F1-score of 11.26%. This highlights the limitations of handcrafted features in capturing the complex and diverse patterns present in igneous rock images.

Table 16.

Performance comparison of different feature classification models on igneous rock dataset.

Feature	Accuracy	Macro Precision	Macro Recall	Macro F1 Score	AUC	AUPR
Image feature	0.9149	0.9342	0.9275	0.9290	0.9946	0.9820
Manual feature	0.2553	0.0908	0.1709	0.1126	0.6085	0.2362
All features	0.9362	0.9688	0.9416	0.9512	0.9962	0.9881

The Bold value indicates best value of metrics

Using image features alone resulted in better performance, achieving 91.49% accuracy, 93.42% macro precision, 92.75% macro recall, and a macro F1-score of 92.90%. This demonstrates the strong representational capacity of deep learning models for igneous rock classification.

When combining handcrafted and image features, the model’s performance improved significantly, with accuracy reaching 93.62%, macro precision rising to 96.88%, and the macro F1-score increasing to 95.12%. The AUC and AUPR values also improved to 0.9962 and 0.9881, respectively. This indicates that the integration of handcrafted features provides additional complementary information that enhances the model’s ability to capture fine-grained details and improve classification accuracy.

To address the cost-effectiveness of this fusion strategy in the deep learning era, we quantified the trade-off between accuracy gains and computational overhead. The handcrafted feature fusion layer adds only 0.03% parameters (11 K) and negligible GFLOPs (0.00%), while increasing inference time by 4.84% (2.60ms absolute). Computing the ratio of accuracy improvement percentage to resource increase percentage yields efficiency ratios of 29.98 (parameters), 43,188 (GFLOPs), and 0.20 (time) for the Sedimentary dataset (0.97% accuracy gain from 98.06% to 99.03%), and 65.84, 94,836, and 0.44 for the Igneous dataset (2.13% gain from 91.49% to 93.62%). The exceptionally high parameter and computational efficiency ratios (all exceeding 29) confirm that handcrafted features deliver substantial accuracy gains with minimal model complexity increase, while the time efficiency ratios, though below unity, correspond to only a 2.60ms absolute increase acceptable for offline geological analysis workflows. Notably, the consistently higher efficiency ratios on the Igneous dataset indicate that handcrafted features provide greater marginal value when baseline deep learning performance is lower, demonstrating their particular utility in challenging classification scenarios where domain-specific knowledge complements learned representations.

Conclusion of feature fusion verification experiments

The feature fusion experiments on both the sedimentary rock sub-dataset and the igneous rock dataset demonstrate the effectiveness of combining handcrafted geological features with deep image representations. While deep learning models alone achieve strong performance, the integration of handcrafted features further enhances the model’s ability to capture low-level details and improve classification accuracy. This multimodal fusion strategy is particularly beneficial for fine-grained classification tasks where subtle visual differences between rock subtypes need to be distinguished. The results highlight the importance of leveraging both deep learning and traditional feature extraction techniques to achieve optimal performance in geological image classification.

Model complexity analysis

While the preceding sections have established the superior classification accuracy of HFANet, it is equally important to evaluate the computational resources it requires. To this end, we conducted a comprehensive analysis of model complexity. Table 17 presents a comparison of the number of parameters (Params), floating-point operations (GFLOPs), GPU memory usage, and average inference time for HFANet and the baseline models.

Table 17.

Comparison of model Complexity. The table lists the number of parameters, GFLOPs, GPU memory usage, and average inference time. Inference time was measured on a single NVIDIA A6000 GPU with a batch size of 1.

Model	Params (M)	GFLOPS	GPU memory (MB)	Inference time (ms)
VGG19	139.61	19.63	566.34	5.428
ResNet34	21.29	3.68	102.58	8.541
DenseNet121	6.96	2.90	129.80	30.936
EfficientNet	4.02	0.41	119.00	22.912
Vison transformer	85.81	11.29	427.71	11.535
Swin transformer	27.53	2.98	545.55	18.945
ConvNeXt	27.83	4.46	539.04	14.475
Rocks2Net	7.05	6.13	183.01	80.952
Enhanced_EfficientNet	4.76	0.67	198.35	55.503
TLCA_ResNet	21.41	3.68	376.16	18.342
SENet_ConvNeXt	29.04	4.47	295.94	31.519
HFANet	34.70	49.33	307.81	74.347

The analysis reveals an interesting computational profile for HFANet. Notably, the model maintains a moderate parameter count of 34.7 M, which is comparable to modern architectures like ConvNeXt (27.83 M) and Swin Transformer (27.53 M), and significantly lower than parameter-heavy models such as VGG19 (139.61 M) and Vision Transformer (85.81 M). This moderate parameter footprint suggests that HFANet achieves its superior performance through architectural efficiency rather than parameter redundancy.

However, the model exhibits higher computational intensity, with GFLOPs reaching 49.33G compared to ConvNeXt’s 4.46G and Swin Transformer’s 2.98G. This increased computational load stems primarily from the dual-branch architecture and the bidirectional cross-attention mechanism, which perform intensive feature interactions rather than simply scaling up model capacity. The inference time of 74.35ms reflects this computational complexity.

Regarding GPU memory consumption, HFANet demonstrates a moderate footprint of 307.81 MB during inference, substantially lower than VGG19 (566.34 MB), Swin Transformer (545.55 MB), and ConvNeXt (539.04 MB). While simpler architectures like ResNet34 (102.58 MB) require less memory, HFANet’s memory usage remains well within the capabilities of modern GPUs. Notably, the disparity between high computational intensity (49.33 GFLOPs) and moderate memory consumption indicates that HFANet is computation-bound rather than memory-bound.

Importantly, the moderate parameter count combined with the architectural nature of the computational overhead suggests significant potential for optimization through engineering approaches. Unlike models that achieve performance through parameter scaling, HFANet’s efficiency bottlenecks are primarily in computational operations that can be optimized through techniques such as kernel fusion, memory access optimization, and hardware-specific implementations. This provides a clear pathway for practical deployment optimization while maintaining the model’s superior accuracy.

Interpretability analysis

To better understand how HFANet makes classification decisions—particularly in the context of complex sedimentary rock imagery characterized by subtle texture variations—we conduct an interpretability analysis comparing HFANet to ConvNeXt, the second-best performing baseline in previous experiments. Unlike traditional convolutional models that rely on localized receptive fields, HFANet integrates local and global information through bidirectional cross-attention mechanisms^57–66, providing an inherent pathway for fine-grained interpretability. To examine the internal reasoning of each model, we design a patch-level explainer for HFANet and employ Grad-CAM⁶⁷ for ConvNeXt to generate corresponding visual explanations^68–72.

The custom explainer for HFANet leverages intermediate signals produced during inference. We extract attention weights from the local-to-global cross-attention module, which indicate the relative importance of each image patch in forming the global representation. At the same time, the local feature embeddings refined by cross-attention are used to capture the semantic activation of each patch via their L2 norm. These two signals are normalized and combined to produce a patch-level saliency map that reflects both where the model is focusing and how strongly each region contributes to the final decision. The resulting map is reshaped and upsampled to the original image size and overlaid as a heatmap for visual inspection. In contrast, the Grad-CAM method computes class-specific activation maps by propagating gradients from the final convolutional layer, highlighting the most influential spatial regions for ConvNeXt’s prediction. By aligning the two types of heatmaps with the original input images, we enable a direct comparison of the interpretability and decision strategies exhibited by the two models.

To provide quantitative validation of the interpretability results, we collaborated with an expert geologist to create semantic segmentation annotations for the sedimentary rock subset. The expert annotated geologically significant features using different colors based on their diagnostic importance for rock classification. We then computed the Intersection over Union (IoU) between our attention maps and expert annotations using color histogram-based similarity metrics. The color IoU calculation involves extracting normalized color histograms from both attention maps and expert annotations in RGB color space. The intersection is computed as the sum of minimum values across corresponding histogram bins, while the union is the sum of maximum values. The IoU is calculated as:

32

where and represent the normalized histograms of attention maps and expert annotations, respectively. Our experiments on the sedimentary rock subset achieved an IoU score of 0.6453, indicating substantial agreement between model attention and expert geological knowledge. This score demonstrates that HFANet captures approximately two-thirds of the diagnostic features identified by geological experts, representing significant alignment with professional geological interpretation.

Figure 13 presents the interpretability comparison results for a representative sample of tuffaceous sedimentary rock. The diagnostic features of tuffaceous sedimentary rocks include volcanic glass fragments, crystal debris structure, grain assemblage characteristics, and matrix features^57,58. In the original polarized-light micrograph, the gray-white, irregularly shaped particles with sharp edges represent volcanic glass fragments formed by rapid cooling during volcanic eruptions. The brighter white particles are crystal debris, while the larger gray particles are lithic fragments, and the black portions represent fine-grained matrix⁵⁹. The combination of volcanic glass fragments and crystal debris constitutes the key diagnostic feature, followed by the ternary assemblage of glass fragments, crystal debris, and lithic fragments^60,61.

Fig. 13.

Interpretability comparison on the Tuffaceous Sedimentary Rock.

The HFANet explainer reveals a fine-grained and spatially distributed attention pattern that successfully captures the critical volcanic glass fragments, showing remarkable consistency with expert annotations. Additionally, it identifies both crystal debris and lithic fragment structures, suggesting that the model integrates both structural and textural cues in its classification decision—an approach consistent with expert geological interpretation^58,62. In contrast, the Grad-CAM visualization focuses primarily on the largest crystal debris while failing to detect the volcanic glass fragments—a crucial diagnostic feature for this rock type. This comparison indicates that HFANet captures richer spatial dependencies and exhibits a more comprehensive understanding of discriminative regions, which is particularly valuable for identifying rock types characterized by heterogeneous microstructures.

Figure 14 shows another example from the tuff rock class. The diagnostic features of tuff include large volcanic glass fragments, volcanic clastic components, residual vesicle structures, clastic structural characteristics, and matrix features⁶³. Large volcanic glass fragments serve as the key diagnostic feature, followed by vesicle structure remnants (fracture networks)^64,65, and then volcanic clastic components (combination of glass debris and crystal debris)⁶⁶. The original image contains a large, centrally located mineral fragment with distinct internal fractures, embedded in a relatively fine-grained matrix. Both the HFANet-generated heatmap and Grad-CAM correctly highlight the key large volcanic glass fragments, but the HFANet attention map additionally focuses on fracture structures and clastic components⁶⁸, showing consistency with expert geological annotations. Notably, some peripheral regions of the surrounding matrix also receive moderate attention, suggesting that HFANet considers contextual background in addition to the dominant feature. In contrast, the Grad-CAM heatmap is heavily concentrated within the central grain, displaying a smooth, blob-like focus that covers the core but lacks structural detail. While both methods correctly emphasize the key discriminative region, HFANet demonstrates greater sensitivity to boundary definition and internal heterogeneity—features that are geologically meaningful and often critical for distinguishing volcanic clastic rocks⁵⁷.

Fig. 14.

Interpretability comparison on Tuff Rock.

A particularly noteworthy aspect of HFANet’s interpretability is its ability to focus clearly on mineral grain boundaries, microfractures, cleavage, and foliation structures—geological features that Grad-CAM fails to capture adequately. This capability is geologically significant for several reasons: grain boundaries often contain diagnostic information about rock formation processes and metamorphic history; microfractures and cleavage planes provide crucial insights into stress history and structural deformation⁶⁵; foliation structures are key indicators of metamorphic grade and deformation intensity. By attending to these subtle but geologically meaningful features, HFANet demonstrates an understanding that aligns with expert geological interpretation principles^58,63, where boundary conditions and structural relationships are as important as the mineral components themselves. This comprehensive attention to both compositional and structural features positions HFANet as a valuable tool for automated petrographic analysis.

Discussion

The experimental results demonstrate that HFANet significantly outperforms existing convolutional and Transformer-based models in classifying thin-section rock images across sedimentary, igneous, and metamorphic datasets. This performance advantage can be primarily attributed to the integration of local and global feature extraction, the bidirectional cross-attention mechanism, and the ensemble classification framework.

HFANet addresses several critical limitations of prior models. Conventional convolutional networks often fail to capture long-range dependencies, while Transformer-based models may overlook fine-grained local textures essential for distinguishing between visually similar rock subtypes. By combining a Swin Transformer-based global branch and a DenseNet-based local branch, HFANet enables complementary feature learning. The bidirectional cross-attention mechanism facilitates mutual information exchange between branches, allowing the network to dynamically focus on discriminative regions at multiple scales.

The inclusion of three independently designed classification heads enhances the model’s robustness and decision diversity. This ensemble strategy proves particularly effective in resolving class confusion, especially for rock types with overlapping visual characteristics.

The incorporation of handcrafted geological features, including color, texture, and luminance descriptors, further boosts the model’s performance. While deep learning models are capable of learning high-level features autonomously, handcrafted features offer interpretable and domain-specific information that complements the learned representations.

Interpretability analysis reinforces the practical value of HFANet. Compared to traditional saliency methods, the attention-based explanation mechanism provides more detailed and geologically meaningful visualizations, highlighting critical mineral boundaries, grain structures, and fracture zones that align with expert geological interpretation.

However, this performance advantage comes with increased computational complexity. While HFANet maintains a moderate parameter count (34.7 M), memory footprint (307.81 MB) and GPU usage(307.81 MB), the high GFLOPs (49.33G) and inference time (74.35ms) stem from the sophisticated dual-branch architecture and intensive cross-attention operations. This computational profile suggests that HFANet is well-suited for applications where accuracy is paramount, such as offline geological research and detailed petrographic analysis, while deployment in real-time or resource-constrained environments would benefit from optimization techniques such as knowledge distillation or operational acceleration.

Conclusions

In this study, we proposed HFANet, a hybrid local-global feature attention network designed to improve the classification of thin-section rock images. By integrating a Swin Transformer-based global branch and a DenseNet-based local branch with bidirectional cross-attention mechanisms and an ensemble classification framework, HFANet effectively captures both semantic context and fine-grained textures. Experimental evaluations across sedimentary, igneous, and metamorphic rock datasets demonstrate that HFANet consistently outperforms state-of-the-art models in all major classification metrics. The incorporation of handcrafted geological features further enhances discriminative power, and interpretability analyses confirm that HFANet provides meaningful and expert-aligned visual reasoning. Future work will focus on expanding HFANet to multi-modal geological datasets, developing optimized variants to reduce computational complexity, and exploring domain adaptation techniques for broader applicability.

Author contributions

Peiyang Wei: Software, Visualization, Investigation, Formal analysis, Funding acquisition, Writing – original draft, Writing – review & editing. Changyuan Fan: Funding acquisition. Xiwen Yang: Conceptualization, Methodology, Formal analysis, Investigation, Supervision, Writing – original draft, Writing – review & editing. Zhibin Li: Formal analysis, Software, Visualization, Investigation, Methodology, Validation, Writing – original draft, Writing – review & editing, Funding acquisition. Ji Zhang: Software, Visualization, Investigation, Writing – original draft, Writing – review & editing. Jianhong Gan: Conceptualization, Methodology, Formal analysis, Investigation, Supervision, Project administration, Software, Validation, Visualization. Xun Deng: Formal analysis, Investigation, Project administration, Software, Validation.All authors reviewed the manuscript.

Funding

This research was funded by the Sichuan Science and Technology Plan: Research and Application of Key and Equipment for Hail Technologies of Command Suppression in Xinjiang (2024YFHZ0151), Sichuan University students innovation and entrepreneurship training program (S202510621042), Chengdu University of Information Technology key project of education reform (JYJG2024206), Key Laboratory of Remote Sensing Application and Innovation (LRSAI-2025004), the open project of Dazhou Key Laboratory of Government Data Security under Grant (ZSAQ202501, ZSAQ202502, ZSAQ202505, ZSAQ202507).

Data Availability.

The image data used in this study were obtained from publicly available datasets. The ‘Photomicrograph dataset of rocks for petrology teaching at Nanjing University’ is available in the Science Data Bank repository, accessible via the persistent identifier 10.11922/sciencedb.j00001.00097. Additionally, the ‘Micro image data set of some rock forming minerals, typical metamorphic minerals and oolitic thin sections’ containing 1446 microscopic images of 11 mineral types is available in the Science Data Bank repository, accessible via the persistent identifier 10.11922/sciencedb.j00001.00014.

Data availability

The image data used in this study were obtained from publicly available datasets. The ‘Photomicrograph dataset of rocks for petrology teaching at Nanjing University’ is available in the Science Data Bank repository, accessible via the persistent identifier https://doi.org/10.11922/sciencedb.j00001.00097. Additionally, the ‘Micro image data set of some rock forming minerals, typical metamorphic minerals and oolitic thin sections’ containing 1446 microscopic images of 11 mineral types is available in the Science Data Bank repository, accessible via the persistent identifier https://doi.org/10.11922/sciencedb.j00001.00014.

Declarations

Competing interests

The authors declare no competing interests.