Hyperspectral data set of stone cultural relics in high-precision machine vision scene

Abstract

In the contemporary era, which is characterized by its emphasis on intelligence and technological advancement, the rapid evolution of machine vision technology has significantly enhanced the efficiency of artificial systems and yielded noteworthy achievements across numerous domains. Nevertheless, in certain specialized fields, such as the high-precision identification of the deterioration modes of stone cultural relics, the limitations of this technology persist. The primary rationale for this limitation is the lack of multi-dimensional and high-precision data support. In response, our team has collaboratively developed hyperspectral acquisition equipment with professional manufacturers and successfully established the world’s first hyperspectral dataset of stone cultural relics (HD-SC-1). This dataset encompasses 28,193 hyperspectral layers from China’s Dazu Rock Carvings. The spectral range of the dataset spans from 400 to 1000 nm, with an image resolution of 2048 × 2046 pixels. The dataset expands the scope of data on stone cultural relic deterioration and enriches the data resources for the protection of stone cultural relics, thereby revitalizing the conservation and restoration of world cultural heritage.

Background & Summary

During the 19th century, there was growing awareness of the need to protect cultural relics and historical structures¹. In the 20th century, the protection of cultural relics gradually transcended national boundaries, evolving into a global concern². In the 21st century, technological advancements have given rise to innovative methods and mechanisms for cultural relics protection^3–6. In recent years, the standards for cultural protection have grown increasingly stringent. As a result, the responsibility for safeguarding these treasures has shifted from traditional caretakers such as temples, aristocrats, and privileged classes to professional archaeologists, historians, and scientific researchers. In the modern era, the widespread integration of digital technology has significantly transformed protection methods, emphasizing efficiency and precision⁷. The successful implementation of these advanced protection measures, defined by their efficiency and precision, relies on the availability of high-quality datasets and sufficient data volume.

The construction and utilization of cultural heritage datasets represent a complex task, fraught with numerous challenges^8–11. To address these challenges, the adoption of advanced technology is essential for ensuring the continuous protection and inheritance of cultural heritage. This process requires not only the meticulous collection of accurate data and the implementation of stringent data management protocols but also the ongoing innovation of methodologies and technological frameworks.

The scope of data pertinent to cultural relics protection is extensive, encompassing subjects ranging from a single historical painting to an entire historical building complex. All such entities fall within the domain of cultural protection. Dazu stone carving is an important part of it.

The cultural heritage dataset in question encompasses a variety of data types, each differing in structure^12,13. In recent years, an increasing number of cultural institutions and enterprises have progressively made their datasets publicly available, fostering the exchange of cultural heritage data and improving the efficacy and success rate of cultural relics protection efforts¹⁴.

Hyperspectral imaging technology integrates the principles of optical imaging and spectral analysis, resulting in the creation of a three-dimensional dataset comprising spatial and spectral information. It is widely used in various disciplines and fields, including agriculture, biomedicine, industrial inspection, and cultural relics preservation^15–19. Compared to traditional RGB imaging, the hyperspectral image offers the following advantages:

1. Fusion of spectral information: The dimension of image information is enhanced, thus improving the accuracy of classification^20–22.

2. The difference of wave reflectivity: Strengthen data analysis from the perspective of material composition^23,24.

The task of our team is to establish a hyperspectral cultural heritage data set and develop an automatic, fast, and accurate recognition model^5,25. The automation of machine vision model training using hyperspectral datasets reduces the reliance on manual intervention^26–28. The use of advanced algorithms and cutting-edge image preprocessing technologies has greatly enhanced the model’s recognition speed^4,29–31. The improved precision in recognition capabilities is due to the cutting-edge equipment developed by our team, including hyperspectral acquisition instruments. These tools enable expanded data coverage, thereby enhancing recognition accuracy and the ability to capture fine details^6,32,33.

In this groundbreaking study, we have successfully developed the first publicly available hyperspectral data set of stone relics. This initiative offers two main contributions: firstly, it enriches the existing dataset of stone cultural relics; secondly, it validates the effective integration of hyperspectral data with the preservation of stone cultural relics. This validation highlights the high-efficiency application potential of hyperspectral imaging in the field of cultural relics protection. Furthermore, this study has broadened the application scope of hyperspectral technology, advanced the development of classification algorithms based on hyperspectral images, and provided substantial technical support for cultural relics preservation. As a result, these achievements have significantly bolstered confidence and enthusiasm in the field of cultural heritage protection.

The structure of this paper is as follows: Section 2 reviews the construction and detection of datasets in image classification. Section 3 details the generation process of the datasets in this study. Section 4 discusses the application outcomes and experimental results of the dataset. Finally, Section 5 provides a comprehensive summary of the dataset’s advantages and potential application value.

Methods

Equipment introduction

To more accurately capture the details of deterioration, our team has collaborated with enterprises to develop a staring hyperspectral spectrometer (SHIS-N220). The instrument is divided into two acquisition systems: an indoor system and an outdoor system. The outdoor system includes a hyperspectral staring camera, a fixture, a tripod, an acquisition computer, and other components. The power supply mode uses an external power supply for the acquisition equipment, and the camera acquisition angle is adjusted manually using a knob and a joystick.

The indoor acquisition system consists of a hyperspectral staring camera, an indoor system console, and acquisition equipment, such as a computer. The power supply method is divided into two parts: the console utilizes residential electricity, while the spectral camera is powered by the acquisition equipment. The camera acquisition angle typically adopts an overhead shot, and the camera distance and the brightness of the light are controlled by the console button, as illustrated in Fig. 3.

Fig. 3.

Hyperspectral data acquisition, correction, and machine vision application flow chart. (a–d) Outdoor acquisition system; (a) Hyperspectral camera; (b) Fixator knob; regulate loose and tight; (c) Fixer handle, adjust the camera angle; (d) Tripod. (a–g) Indoor acquisition system; (e) Indoor system console - simulating light sources; (f) Camera telescopic frame; (g) Console base, control camera moving up and down and light switch. An) Information to be tested after correction; Bn) identification result; n = 1, 2, 3, 4. DET: Detachment; FIML: Features Induced by Material Loss; DD: Discoloration & Deposit; BC: Biological Colonization. Where Ι is the corrected reflectance, R₀ is the original reflectance, R_D is the dark reference spectrum, and R_w is the white reference spectrum.

Data correction

Despite the spectrometer’s exceptional performance, it is subject to noise due to the impact of dark current during the acquisition process, which compromises the quality of the resulting data. The quality of ambient light also significantly influences the acquisition of images. To address these challenges, a whiteboard with a reflectivity of 99.99% is employed as a reference to neutralize the effect of ambient light and dark shooting correction is implemented to counteract the influence of dark current. It is imperative to note that this correction process must be executed individually, culminating in the formation of a comprehensive dataset, as illustrated in Fig. 3.

Shooting challenges

The quality of the data obtained is contingent upon the limited protection status and access. Furthermore, the large number of tourists results in some data lacking a suitable shooting angle. During peak travel periods, some data lacks the corresponding environmental light, which poses a challenge to data collection. Nevertheless, the collection of high-quality data is prioritized.

Data Records

Presently, the public can access the hyperspectral dataset of stone relics through 10.6084/m9.figshare.28606919³⁴. The system’s automated process for uploading files cancels folder attributes, resulting in a simpler naming method for more intuitive viewing of the uploaded files. The constructed dataset is multimodal, containing various data types, including spectral, image, spatial, color, and text data. It also includes images (.img/.tif), text (.md/.xlsx), and feature binary datasets. The datasets are divided into three separate folders according to content, as shown in Fig. 1.

Fig. 1.

Dataset Hierarchical Architecture Diagram.

The dataset exhibits a high degree of flexibility, supporting spectral data-based processing, image-based processing, and comprehensive processing of spectral and image combinations. The specific application mode can be adjusted according to the user’s research objectives, thereby meeting diverse research needs.

Data overview

The Dazu Rock Carvings were first constructed in the early Tang Dynasty and continued over several dynasties until the Ming and Qing Dynasties. Currently, 75 cultural relics protection grottoes are dispersed throughout 24 towns and streets in Dazu District. The Dazu stone carvings are regarded as the archetype of late grotto art in China, possessing invaluable historical, artistic, and scientific significance. In 1999, they were inscribed on the World Heritage List by the United Nations Educational, Scientific and Cultural Organization (UNESCO).

The reclining Buddha of Baoding Mountain (Sakyamuni Nirvana Scene) is an important component of the Dazu stone carvings, with a body measuring 31 meters in length, 3.2 meters in width, and 6.8 meters in height. The carving’s history can be traced back 851 years. A parallel can be drawn between the Dazu Baoding Mountain Buddha and the renowned Angkor Wat, as both sites share similar monsoon climate characteristics, topographical proximity, and comparable vegetation and ecosystem features. The deterioration patterns exhibited by the Baoding Mountain reclining Buddha closely mirror those observed in Angkor Wat, thereby validating the selection of Dazu Baoding Mountain as a representative sampling site. As shown in Fig. 2.

Fig. 2.

World famous grottoes environment similarity research map. Data sources³⁵, download on https://www.worldclim.org/data/worldclim21.html. The Angkor Wat image data comes from the network. Average temperature (°C): The average temperature in January from 1970 to 2000. (Download number of the website: tavg 10 m-10 minutes). Precipitation (mm): The average precipitation in January of each year from 1970 to 2000. (Download number of the website: prec 10 m-10 minutes). Bioclimatic variables: The average value of each variable in January for the years 1970–2000. (Download number of the website: bio 10 m-10 minutes). Elevation: The data comes from SRTM elevation data. (Download number of the website: elev 10 m-10 minutes).

To provide a more comprehensive dataset on the deterioration of stone relics, we collected panoramic spectral data of the Buddha (Sakyamuni Nirvana Scene) at the site and sorted the data according to deterioration modes. In addition, we invited relevant staff from the Dazu Rock Carving Research Institute to participate.

The dataset under consideration contains a total of 28,193 spectral information layers, 233 visible light high-definition layers, 233 visible light spatial features, 466 visible light color spaces, and simple deterioration classification tables. The spectral wavelength range extends from 400 to 1000 nm, with a wavelength interval of 5 nanometers. The resolution is 2048 × 2046 pixels, and the adaptation algorithm is designed for classification, such as Att-RBF-RF⁴.

Data statistics

We chose the Dazu Rock Carvings as the first sampling point, using it as a starting point to gradually promote our hyperspectral dataset plan nationwide and eventually worldwide. The primary principle in creating the dataset is to represent the deterioration modes of stone cultural relics (the data format is strictly in accordance with the ENVI standard). Among them, discoloration and deposit, as well as features induced by material loss and detachment, are more prevalent.

The preliminary classification and collation of the dataset have been completed. The dataset spans 730 days, encompassing 233 high-resolution hyperspectral cubes, which collectively cover 28,193 independent layers. Of these, 11,616 layers are directly related to the statue, while 16,577 layers exhibit a weaker association. Additionally, the dataset meticulously documents five distinct deterioration modes. It is important to acknowledge that, due to the complexity of the environment, these deterioration patterns are interconnected and not entirely independent, which explains why the total number of deterioration pattern layers exceeds the total number of individual layers. To facilitate a comprehensive understanding and analysis of these deterioration patterns, we have provided a set of text datasets as a classification reference. Fig. 4 illustrates the structure and composition of the dataset in detail, providing a clear overview of the research.

Fig. 4.

Dataset Classification and Deterioration Mode Classification Diagram. The above figure represents the classification details of the data layer structure, where the x-axis represents the acquisition time, and the y-axis represents the number of layers. The following diagram shows the data deterioration classification summary. Given the complexity of the collection environment, it is reasonable to categorize the data based on the presence or absence of statues. Hyperspectral data is inherently a three-dimensional cube, which naturally includes some redundancy. Therefore, it is logical to adopt a layered classification approach to manage and analyze the data efficiently. The term ‘statue’ refers to a general category containing image information such as Buddha heads or statues. Specifically, if the layer data contains a Buddha image, it will be classified as a statue category; otherwise, it will be classified into a different category.

Technical Validation

Deterioration pattern classification

In this section, the HD-SC-1 dataset, proposed by the authors, is utilized for the training and evaluation of a classification model. The purpose of this model is to identify and evaluate deterioration patterns. Training the model generally involves several key steps, including the selection of the dataset, annotation, coding, fine-tuning, and recording.

In identifying more complex regional deterioration patterns, the selection of an appropriate dataset for model training is of particular importance. In comparison with efficient algorithms, high-quality datasets play a more significant role in improving model performance. Therefore, in model training, particular attention is paid to the degree of matching between the dataset and the training task, as well as the quality of the dataset, to ensure the accuracy of the training results and the reliability of the model. This strategy facilitates the construction of a more advanced recognition model and effectively enhances the accuracy of deterioration pattern recognition. It is noteworthy that the data feature set is derived entirely from raw hyperspectral data.

As illustrated in Fig. 5, the process is delineated in meticulous detail, from the initial dataset preparation to the training and evaluation of the model. This methodical approach is meticulously designed to ensure the efficient training of models capable of accurately identifying deterioration patterns while preserving the simplicity and efficiency of the operational process.

Fig. 5.

Workflow Diagram for Data Acquisition, Processing, and Classification Training. Data distillation is a conventional method to improve the efficiency and accuracy of model operation. This paper makes full use of the existing information for model training. An) Information to be tested after correction; Bn) identification result; n = 1, 2, …, 5. The rightmost column is a confusion matrix that evaluates the accuracy of the model classification. CD: Crack & Deformation; DET: Detachment; FIML: Features Induced by Material Loss; DD: Discoloration & Deposit; BC: Biological Colonization.

The specific experiment was conducted using the following environment:

CPU: Intel(R) Xeon(R) E.

GPU: Quadro-p5200.

Operating System: Linux.

Frameworks and Libraries: TensorFlow 2; OpenCV 2.

Programming Language: Python 3.9.

To facilitate a more thorough examination of the deterioration modes, this study builds upon the prior research of the research team. The combination of 475 nm, 520 nm, 650 nm, 670 nm, 745 nm, and 750 nm is selected to form a true color map, which can more effectively identify the deterioration modes. The algorithm flow diagram is depicted in Fig. 6. The quality of the collected dataset is then assessed through a random sampling method, with the outcomes of this assessment incorporated into Fig. 7.

Fig. 6.

Algorithm Logic Diagram. A simple description of the process of the classification algorithm.

Fig. 7.

Deterioration Mode Test Integration Diagram. CD: Crack & Deformation; DET: Detachment; FIML: Features Induced by Material Loss; DD: Discoloration & Deposit; BC: Biological Colonization.

Experimental result analysis

To quantitatively evaluate the detection accuracy of the model, we used the F1 Score and Kappa Coefficient. These evaluation indicators provide a more accurate measurement of the model’s performance. The specific calculation formulas are as follows:

$$F_1\mathit{Score}=\frac{2\mathit{Precision\; Recall}}{\mathit{Precision}+\mathit{Recall}}\le 1$$ 1

$$K\mathit{appa}=\frac{P_0-P_e}{1-P_e}$$ 2

In formulas (2):

P₀: represents the observed agreement, P_e: represents the expected agreement.

These metrics help us evaluate the model’s accuracy in identifying deterioration patterns in stone cultural relics more comprehensively.

The findings indicate that the model demonstrates a satisfactory capacity to identify the deterioration modes of stone cultural relics. The specific outcomes are enumerated in Tables 1, 2, and Fig. 7.

Table 1.

Identification model evaluation index record table.

Image	O1	O2	O3	O4	O5	O6	O7	O8	O9	O10	O11
Kappa	0.67	0.90	0.86	0.94	0.94	0.80	0.93	0.98	1.00	1.00	0.97
F1	0.78	0.93	0.93	0.97	0.97	0.90	0.97	0.99	1.00	1.00	0.99
$\stackrel{\circledR }{F1}$						0.95

Note: The data is presented to preserve 2 decimal places for beauty, and 4 decimal places are reserved for accuracy during calculation.

Table 2.

Reference Research Indicators Record Sheet.

Model	SVM	RF	CTB	LGBM
$\stackrel{\circledR }{\stackrel{\circledR }{F1}}$4	0.90	0.94	0.96	0.91
$\stackrel{\circledR }{\stackrel{\circledR }{F1}}$	0.93

Note: The data is presented to preserve 2 decimal places for beauty, and 4 decimal places are reserved for accuracy during calculation.

Considering the heterogeneity among identification models documented in prior studies, this study employs the $\stackrel{\circledR }{F1}$ metric as the evaluation criterion. It is acknowledged that the performance of different models may vary, and direct comparisons may be limited. Nevertheless, it is hypothesized that the $\stackrel{\circledR }{F1}$ score can serve as a reliable indicator of overall research quality to a reasonable extent. To this end, a quadratic average of the indicators obtained from the study was conducted and then compared with the indicators of the model in this study. This methodological approach ensures the comprehensiveness and representativeness of the evaluation.

An examination of the numerical indicators reveals that there are no significant differences between the models, suggesting that the models and datasets employed possess effective recognition capabilities. This result provides substantial evidence for the reliability of the model and the applicability of the dataset, thereby confirming the validity of our methodological approach and the scientific rigor of the research method.

Usage Notes

All uploaded hyperspectral data are rigorously verified to ensure their accuracy and reliability, and readers can download them directly. In terms of spectral data processing, commonly used methods include programming processing and professional software processing. Among them, programming processing is often realized by professional data packets such as Spectral Python, while software processing uses professional tools such as ENVI. It is assumed that the algorithm used has excellent performance. To further improve the fineness of the recognition effect, it is necessary to combine spectral information and spatial information to construct a more accurate recognition model.

Author contributions

Haiqing Yang: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Funding acquisition, Writing - original draft. Lixin Peng: Formal analysis, Investigation, Methodology, Validation, Visualization, Writing - original draft. Xingyue Li, Chiwei Chen, Shide Liu, Gang Zhao: Conceptualization, Data curation, Methodology, Validation, Writing - review & editing.

Data availability

Presently, the public can access the hyperspectral dataset of stone relics through 10.6084/m9.figshare.28606919. Please quote this article when you use it.

Code availability

The corresponding code and software are exposed to 10.6084/m9.figshare.28606919. In addition, you can also directly contact the author to obtain.

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.