A Multimodal Railway Vibration and Vision Dataset (Rail-VIVID)

Abstract

Indirect, onboard monitoring of railway track infrastructure is critical for improving safety and operational efficiency, yet its adoption remains limited due to the complexity of real-world data. Development has historically followed two separate paths: vibration analysis, which offers high sensitivity to subsurface anomalies but lacks spatial context, and vision-based inspection, which provides spatial context but is constrained by environmental conditions and limited to surface-level detection. Progress in both approaches—especially their integration—has been limited by the lack of a publicly available benchmark dataset for controlled-condition analysis, including multimodal cases. To address this gap, we present a novel temporally synchronized railroad vibration and vision dataset (Rail-VIVID). This dataset, collected from repeated runs over a track segment with documented anomalies under controlled conditions, serves two main goals. It enables investigation into the relationship between track anomalies and their signatures in each sensing modality by serving as a benchmark for anomaly detection algorithms. It also provides a foundation for developing integrated, multimodal sensor fusion approaches to support more robust detection and predictive maintenance.

Background & Summary

The transition to onboard, indirect monitoring for railway safety—where sensors installed on railroad vehicles collect data to detect track defects—represents a long-standing goal within the transportation industry, promising a paradigm shift from periodic, manual inspections to continuous, automated assessment¹. Historically, the development of these systems has proceeded along two largely separate tracks: vibration-based analysis and vision-based inspection.

The first approach, utilizing sensors like accelerometers installed onboard vehicles, measures the dynamic response of the vehicle as it interacts with the track^2–5. Its primary strength is its sensitivity to the underlying physical responses that indicate track irregularities, including subsurface defects. However, this approach is hindered by a fundamental challenge: the relationship between the measured signal and the physical track anomaly is indirect. Onboard sensors measure the vehicle’s dynamic response to an anomaly, not the anomaly itself, making the extraction of a defect’s characteristic signature a complex inverse problem. This core difficulty is then compounded by the operational environment. The variable, and often harsh, conditions—extreme vibrations, temperature fluctuations, and electromagnetic interference—introduce significant noise, resulting in a low signal-to-noise ratio that further masks the subtle anomaly signatures⁶. Furthermore, the inherent variability of in-service operations, where factors such as vehicle speed and axle load are in constant flux, adds another layer of complexity, making it difficult to distinguish a true anomaly from benign operational changes⁷.

The second approach, employing onboard cameras to support indirect vision-based analysis, provides clear spatial context by identifying visible, surface-level defects like broken fasteners or rail-head damage^8–11. However, this method faces its own set of challenges that limit its real-world reliability. Its effectiveness is highly dependent on external conditions and can be severely compromised by poor lighting, adverse weather, and track debris. More fundamentally, it is incapable of detecting subsurface flaws that may represent a more significant structural risk. The respective limitations of each approach—the signal complexity and indirectness for vibration, and the environmental sensitivity and surface-level dependence for vision—mean that many attempts to develop robust, single-modality systems have struggled to move beyond controlled laboratory settings^12–15 into reliable industrial practice¹.

A key barrier preventing the transition of both approaches from research to practice is the lack of publicly available, full-scale benchmark datasets¹⁶. Creating such a dataset is a significant undertaking, which helps explain why none currently exist. It requires not only access to railway infrastructure—a substantial logistical and financial challenge—but also the deployment of an advanced multimodal sensing system capable of capturing synchronized, high-fidelity data under variable operational conditions⁶. Furthermore, the most critical component—the ground truth—requires a meticulous, labor-intensive survey to precisely locate and comprehensively document every anomaly along the track. This combination of high logistical barriers, complex instrumentation requirements, and the demanding process of creating reliable ground truth has made the generation of such datasets prohibitively difficult¹⁷.

Currently, the few existing datasets related to railway maintenance that are publicly available are collected from fixed wayside inspection systems¹⁸. While valuable for periodic structural assessment and detecting surface-level defects as trains pass, these systems cannot capture the complex vehicle-track dynamics that are essential for developing and validating onboard solutions. The few datasets that are collected from moving vehicles have been gathered for other purposes, such as monitoring the health of the vehicle itself, or are recorded in uncontrolled operational environments^19,20. This lack of systematic, ground-truth data makes it extremely difficult to rigorously study the fundamental relationship between specific track anomalies and their corresponding sensor signatures, or to reliably benchmark the performance of detection algorithms. Crucially, it also impedes emerging efforts to integrate vibration and vision data into unified, multimodal systems.

Recent advances in sensor fusion and machine learning offer new opportunities to overcome the limitations of single-modality systems^21–23. The two primary approaches to onboard monitoring are inherently complementary: the strengths of one directly address the weaknesses of the other. Vibration analysis can detect subsurface issues where vision fails, while vision provides the crucial ground-truth context and precise localization that vibration data inherently lacks²⁴. This reciprocal relationship makes it reasonable to hypothesize that a unified, multimodal system could achieve greater robustness, reliability, and diagnostic completeness than either modality alone.

The fundamental challenge of identifying subtle defect signatures within complex data streams is not unique to railways but is representative of a wider class of anomaly detection problems found in large-scale engineering and infrastructure systems. In these applications, an anomaly’s signature is rarely a simple, isolated event. Instead, it is characterized by a distinctive correlation between time-series signals, their frequency content, and the physical location of the sensors that capture them²⁵. For example, in aerospace engineering, detecting structural fatigue in an aircraft’s wing requires correlating minute changes in vibration patterns across a distributed network of sensors²⁶. Likewise, in civil infrastructure, identifying localized damage in a bridge involves analyzing how its natural frequencies and mode shapes, captured by an array of accelerometers, shift in response to excitation²⁷. In each of these domains, the core task is to develop algorithms capable of learning these complex, multi-faceted signatures. Therefore, this dataset, with its multi-channel vibration data, serves not only as a benchmark for railway applications but also as a canonical case study for developing and validating advanced signal processing and machine learning techniques that can be generalized to a broader class of critical infrastructure monitoring problems.

To address the critical lack of benchmark data for both single-modality development and emerging multimodal approaches, we present a novel temporally synchronized railroad vibration and vision dataset (Rail-VIVID), collected onboard a full-scale railroad vehicle. The dataset was generated from 20 repeated runs over a fixed 1.4-km segment of railway track containing a comprehensive set of nine documented, ground-truthed anomalies. All data were collected under a matrix of controlled operational conditions—including varying vehicle speeds—and are supplemented with synchronized environmental data.

The primary goal of this dataset is twofold. First, it provides a high-quality benchmark for the development, validation, and comparison of single-modality algorithms—both vibration- and vision-based. Second, it serves as a foundational resource for advancing research on integrated multimodal systems that intelligently unify these complementary data streams. By making this dataset publicly available, we aim to accelerate the shift from reactive to proactive maintenance and support the development of more robust and reliable infrastructure monitoring solutions²⁸.

Methods

Experimental Design

The experiment was conducted on a test track at the East Broad Top Railroad in Rockhill Furnace, PA, as shown in Fig. 1. The total track is 2.0 km (marked in red in Fig. 1) and the segment that the controlled tests were conducted on is 1.4 km (marked in green in Fig. 1). This site hosts demonstration and testing operations for the vehicle used in this study, providing a controlled environment ideal for repeated data collection runs. The main objective of the experiment was to collect synchronized, multimodal sensor data—specifically high-frequency acceleration and high-resolution video—over a fixed track segment containing exhaustively known, naturally occurring anomalies under a matrix of varied, yet controlled, operational conditions.

Fig. 1.

(a) Schematic and (b) satellite image of the experimental test track. The red line indicates the full testbed route, while the green line marks the 1.4-km segment used for data collection. Start and end points are highlighted, and pre-surveyed locations of naturally occurring track anomalies are marked with cones. Orange cones indicate rail joint anomalies, while black cones indicate divergence and convergence points.

The test vehicle was a two-car Pop-Up Metro Class 230 light metro train²⁹, with one car instrumented for data collection, as shown in Fig. 2. This train is a newly built conversion of former London Underground D78 Stock, reusing the original aluminum body shells and bogies. The vehicle is propelled by a battery-electric multiple unit system capable of regenerative braking, making it part of a new generation of rolling stock designed for efficiency and operational adaptability.

Fig. 2.

The instrumented Class 230 battery-electric test vehicle. Top: Schematic view of the sensor layout and key dimensions of the vehicle. Bottom: View of the vehicle at the test site from different directions. View ⓐ shows the front of the vehicle when traveling from Point B to Point A, ⓑ shows the front of the vehicle when traveling from Point A to Point B, and ⓒ shows the vision system.

To ensure spatial accuracy and reliable ground-truthing, a rigorous calibration protocol was implemented. Cones were placed at the precise, pre-surveyed start and end points of the test segment (referred to as “Point A” and “Point B,” respectively) and at the location of each documented anomaly, as shown in Fig. 1. These static markers served as consistent physical references during each run, visible both in the vision system data and to the operators. The ground-truth anomalies, specifically those related to railroad fasteners and rail joints, were manually characterized. Key dimensions of each anomaly were measured multiple times to ensure consistency, and the average values were recorded.

The experimental design was structured as a 2 × 3 factorial design to systematically evaluate the influence of two primary factors: direction of travel and vehicle speed. The direction factor had two levels: uphill (Point A to Point B) and downhill (Point B to Point A), with an elevation difference of 8.88 m. The speed factor had three nominal levels: slow, medium, and fast. However, due to the grade, the specific speeds for each level were direction-dependent:

• Uphill (Point A to Point B): Slow (20 km/hr), Medium (22.5 km/hr)

• Downhill (Point B to Point A): Slow (22.5 km/hr), Medium (25 km/hr), Fast (27.5 km/hr)

During preliminary testing, a practical operational constraint was identified: the vehicle could not safely and consistently achieve the fast speed level in the uphill direction due to locomotive power limitations. Rather than redefining the fast speed to match the maximum achievable uphill speed—which would have constrained the downhill operations—or dropping the fast downhill runs to enforce an identical experiment setting across directions, the decision was made to retain all collected data. This results in five distinct experimental groups rather than a fully balanced 2 × 3 factorial design. To ensure statistical robustness, four replicate runs were collected for each condition, yielding a total of 20 runs in the final dataset. This design allows for rigorous analysis of the main effects of speed and direction while acknowledging real-world operational boundaries. Future research that requires a symmetric experiment design may exclude the fast downhill condition.

To ensure operational control and data quality, the vehicle’s speed was managed by the operator using a built-in speedometer and a control lever to maintain the target speed for each run. The consistency of the speed was then independently verified during post-processing using the high-resolution localization data.

Data Acquisition Design

The onboard data acquisition system is composed of three primary subsystems: a vibration sensing system, a vision sensing system, and a localization system. All subsystems interface with a central processing laptop that handles data post-processing, as depicted in the system architecture schematic in Fig. 3. The entire instrumentation suite was powered by a dedicated battery system to isolate it from the vehicle’s electrical systems and minimize potential electromagnetic interference.

Fig. 3.

A high-level schematic of the complete data collection system, illustrating the parallel vibration, vision, and localization subsystems and their connection to the central processing laptop.

Vibration Sensing System

The vibration sensing system comprises six Silicon Designs Model 2012 1D piezoelectric accelerometers³⁰. The ±5g range model is commonly used in state-of-the-art instrumentation for low-speed vehicles (below 96.56 km/hr)^31–34. Here, g = 9.8 m/s² denotes the gravitational acceleration. It was selected for its high sensitivity to the low-amplitude vibrations relevant to track monitoring, while also avoiding saturation during larger shock events. This sensor offers a typical frequency response at a −3 dB bandwidth of 800 Hz, though the effective bandwidth may be extended through frequency response compensation (e.g., up to 1 kHz); per the Nyquist-Shannon sampling theorem, which requires the sampling rate to be at least twice the maximum frequency of interest to prevent aliasing, a 2 kHz sampling rate is therefore a safe choice, and is the rate adopted in this study. As shown in Fig. 4, the six sensors were distributed across the vehicle to capture the propagation of vibrations through the suspension systems: two sensors were mounted on the axles (Channel 2 & Channel 4 in Fig. 4) to measure unsprung mass dynamics at the wheel-rail interface; two were mounted on the bogie frames (Channel 1 & Channel 3 in Fig. 4) to capture vibrations after the primary suspension stage; and two were mounted on the car body (Channel 5 & Channel 6 in Fig. 4) to measure the final, attenuated vibrations experienced within the vehicle.

Fig. 4.

Sensor installation and layout on the test vehicle, highlighting the placement of the sensors (channels) on the vehicle’s bogies, axle boxes, body, and front/end.

The analog signal from each of the six accelerometer channels is routed through a custom-built signal conditioning circuit prior to digitization. This circuit consists of two main stages. First, an AD623A Instrumentation Amplifier provides low-noise amplification of the sensor’s differential output signal³⁵. The amplified signal is then passed through a MAX7401, an 8th-order Bessel low-pass filter, which serves as an anti-aliasing filter³⁶. A Bessel filter topology was specifically chosen for its linear phase response, which is critical for preserving the temporal shape of transient vibration events. The conditioned analog signals are then digitized at 2 kHz with 16-bit resolution by an NI cDAQ-9185 chassis³⁷. An Arduino UNO R3 board is used in the circuit solely to provide a stable 5V power supply to the accelerometers, isolating their power source from the primary data acquisition hardware³⁸. This digital data is subsequently transmitted from the NI chassis to the central laptop via a single Ethernet connection.

Vision Sensing System

The vision system, installed 1.52 m above the sleepers and angled 15 degrees downward as shown in Fig. 2, consists of a See3CAM 50CUGM industrial camera. This unit features a 5-megapixel SONY IMX264LLR-C monochrome CMOS sensor with a global shutter, which is critical for capturing sharp, motion-blur-free images from a moving platform³⁹. The camera was configured to stream high-resolution (1920 × 1080 pixels) video. While the system was configured for a target frame rate of 100 frames per second (FPS), the effective frame rate during data logging was typically between 30-40 FPS due to input/output bottlenecks of the logging hard drive. This achieved frame rate was still sufficient to ensure a high degree of spatial overlap between consecutive frames, providing a continuous visual record of the track. The camera is UVC compliant and connects directly to the central processing laptop via a high-speed USB 3.2 Gen 1 interface, which provides both power and data transfer.

Localization System

The Localization System consists of a Columbus P-7 Pro multi-constellation GNSS receiver^40,41. This unit provides location data (latitude and longitude) at an update rate of 1 kHz with a stated horizontal accuracy of 0.3 m, achieved by upsampling and Kalman smoothing to produce positional estimates. It connects directly to the central processing laptop via a USB 3.2 interface. To ensure an unobstructed view of the sky for optimal satellite signal reception, the GNSS antenna was mounted on the roof of the vehicle.

Environmental Data Acquisition

To account for the potential influence of environmental factors on sensor readings, ancillary data was incorporated from a weather station located approximately 40.23 km from the test site. This data, which included variables such as ambient temperature and relative humidity, were recorded at a one-hour resolution. This information provides valuable context for analyzing the primary sensor data across different collection days.

Computational Processing

The computational processing pipeline was designed to convert the raw, multimodal data streams from the data acquisition system into continuous, calibrated, and synchronized files for each valid experimental run. All processing was performed using custom scripts written in Python and are made available within the dataset. A schematic of this entire workflow, from the initial data ingestion to the final integrated files, is illustrated in Fig. 3. The diagram shows the sequence of processing steps and the intermediate data generated at each stage. As shown, the pipeline consists of four main stages.

Initial Data Ingestion

The raw data from all sensor systems were first ingested. The accelerometer data, initially logged in contiguous 2-second windows to ensure data integrity during acquisition, were stitched together to form a single continuous data stream for each run. The start and end of each run were then programmatically identified by finding the timestamps corresponding to the minimum Euclidean distance between the vehicle’s recorded GNSS coordinates and the pre-defined start and end point coordinates of the test segment. Finally, runs were validated for quality; any run exhibiting sensor data dropouts or inconsistent speed was excluded from the final dataset.

Vibration Data Processing

The raw output of each of the six accelerometer channels was converted from its raw digital units to physical units of gravitational acceleration (g). This conversion was performed using a set of sensor-specific sensitivity and bias values, which are provided in Table 1. Following this calibration, a digital low-pass filter was applied to the acceleration data. A 4th-order Butterworth filter was implemented with a cutoff frequency of 500 Hz. This cutoff frequency was selected to isolate the frequency band where the structural response to short-scale track anomalies is most prominent. The impact dynamics generated at the wheel-rail interface by such defects primarily excite frequency modes within the 0-500 Hz range, while higher frequencies are more characteristic of benign operational noise. The Butterworth filter topology was chosen for its maximally flat frequency response in the passband, ensuring that the amplitude of the anomaly-related signals is not distorted by the filtering process.

Table 1.

Accelerometer parameters and constants.

Channel ID	Bias [mV]	Sensitivity [mV/g]
1 (Bogie, Front)	27.5	800.0
2 (Axle, Front)	17.2	799.6
3 (Bogie, Back)	11.0	797.6
4 (Axle, Back)	28.9	795.2
5 (Body, Front)	−10.3	798.8
6 (Body, Middle)	27.9	796.4

Vision Data Processing

The high-resolution video stream recorded during each run was processed to extract individual image frames. Each frame was saved as a separate image file and associated with its precise acquisition timestamp, provided by the central processing laptop’s system clock. This process converts the continuous video log into a discrete sequence of images, each representing a specific moment in time and a specific location along the track, ready for synchronization with the other data streams.

Final Data Integration

The final step involved integrating and synchronizing all processed data streams onto a common time base. The 2 kHz vibration data served as the primary time index. The 1 kHz GNSS data was upsampled to the same time base using linear interpolation. Similarly, the hourly temperature and humidity readings from the nearby weather station were upsampled to match the 2 kHz timestamps using cubic spline interpolation to create a smooth and continuous estimate of the environmental conditions during the run. Temporal alignment of the image frames with the tabular data was achieved as follows. Since both the camera and the accelerometer system operate on the same central processing laptop system clock, they share a common absolute time reference. The start and end of each run served as anchor events for alignment. In the image stream, these boundaries were identified algorithmically via blur detection and subsequently verified through manual visual inspection; in the acceleration records, they were identified directly from location signals. The fully processed and synchronized multimodal data for each run were then exported as a set of Comma-Separated Values (.CSV) files and associated image folders.

Data Records

The dataset, version v1.0.0, is permanently archived on Hugging Face at https://doi.org/10.57967/hf/8411⁴². The repository is structured into a series of folders, where each folder corresponds to a single experimental run and is named using the convention Direction_Speed_Index. For example, the folder AtoB_20_3 contains all data for the third experimental run conducted from Point A to Point B at a target speed of 20 km/hr. Each run folder contains two primary components: a .CSV file for acceleration data and a folder of images. In addition to the run-specific data, the repository includes a comprehensive record of all documented track anomalies, which serves as the ground-truth for all runs.

Hierarchical Structure of the Dataset

Here, the dataset structure is shown. All other run folders follow the same internal structure.

Tabular Data Record

Within each run folder is a single .CSV file containing the synchronized time-series data from the accelerometers, GNSS, and environmental sensors. The file uses the Unix timestamp as the primary key to link all synchronous measurements. Each .CSV file contains the following columns:

• Timestamp: The Unix timestamp in seconds, with microsecond precision.

• Channel1 through Channel6: The vertical z-axis acceleration data in units of g. The specific physical location for each channel is detailed in Table 1.

• Temperature: The ambient temperature in degrees Fahrenheit (°F), upsampled from hourly readings.

• Humidity: The relative humidity, expressed as a decimal between 0 and 1, upsampled from hourly readings.

• Latitude: The geographic latitude of the vehicle in decimal degrees.

• Longitude: The geographic longitude of the vehicle in decimal degrees.

Vision Data Record

A subfolder named images resides within each run folder. This folder contains the complete sequence of high-resolution image frames extracted from the vision system for that run. Each image is a .JPG file named with the timestamp at which it was captured, with microsecond precision (e.g., 2025-04-11_15-35-10.100178.JPG). This naming convention ensures that each image is uniquely identified and can be directly correlated with the corresponding row in the .CSV file via the shared time base.

Ground-Truth Anomaly Record

A comprehensive record of all documented track anomalies is provided in the repository’s root directory as a separate file. This record details the precise location and physical characteristics (including 3D point clouds) of irregularities within the test segment, serving as the ground truth for algorithm development and validation. The details of these anomalies are summarized in Table 2.

Table 2.

Documented ground-truth track anomalies.

Index	Type	Size [cm]	Latitude	Longitude
1	Joints	2.0	40.23553	−77.88898
2	Joints	2.0	40.23512	−77.88770
3	Joints	2.4	40.23493	−77.88711
4	Divergence	NAN	40.23461	−77.88612
5	Convergence	NAN	40.23425	−77.88485
6	Joints	2.0	40.23388	−77.88320
7	Joints	2.5	40.23242	−77.87795
8	Joints	2.3	40.23205	−77.87651
9	Joints	3.0	40.23194	−77.87508

Together, these components provide a complete, multimodal, and synchronized record of each experimental run, explicitly designed to facilitate the development and benchmarking of advanced data analysis techniques.

Technical Validation

To validate the technical quality of the dataset, a series of analyses were performed to confirm the plausibility of the signals, the reliability of the measurements, and the correlation between the sensor data and the ground-truth track anomalies.

Signal Correlation

A quality check was performed to ensure the integrity of the raw sensor outputs. Visual inspection of the data confirmed that the signals were free from common acquisition errors such as clipping, dropouts, or excessive electronic noise. Figure 5 shows a representative 2-second snippet of the z-axis acceleration from all six channels, illustrating the typical structure of the collected vibration data. To further validate the physical plausibility of the measurements, a correlation analysis was performed between all sensor channels. This analysis serves to confirm that the recorded signals exhibit spatial relationships consistent with their physical placement on the vehicle, as described in the Methods section. The correlation matrix, shown in Fig. 6, was computed by averaging the Pearson correlation of sampled 2-second data segments from multiple runs. In each cell, the top number represents the mean correlation, while the number in parentheses indicates the standard deviation, capturing the variability of the relationship.

Fig. 5.

A representative 2-second time-series segment from all channels, showing the typical vibration signal structure.

Fig. 6.

Pearson correlation matrix for the six accelerometer channels sampled within a 2-second window. Note: Channel is denoted as “Ch.”.

The results align with physical expectations and reveal key dynamic properties of the system. There is a moderate positive correlation (mean of 0.45) between the two sensors mounted on the car body (Channel 5 & Channel 6). This is expected, as these sensors measure the lower-frequency, global motions of the vehicle body, which moves as a semi-rigid unit. Conversely, the correlation between the back bogie and its corresponding axle (Channel 3 & Channel 4) is weakly negative (mean of − 0.12), confirming they capture distinct, localized dynamic events.

A key finding from this analysis is the significant variability in some of these relationships, indicated by the high standard deviation. For instance, the correlation between the two body-mounted sensors (Channel 5 & Channel 6) has a very high standard deviation (0.29). This suggests that while the body sections often move as one entity, their dynamic relationship changes significantly, likely in response to track twists or anomalies. Similarly, the notable standard deviation in the correlation between the front and back bogies (e.g., Channel 1 and Channel 3, standard deviation of 0.21) and between the back bogie and axle (Channel 3 & Channel 4, standard deviation of 0.20) indicates that while these components are largely independent, there are moments where their motions become coupled. Also of note is the distinct effect of engine noise on the front axle (Channel 2).

Anomalies Validation

Another key validation was performed to confirm that the sensor data contains clear signatures of the documented track anomalies. For this analysis, the test track was computationally divided into contiguous 7.62-m segments. Each segment was labeled as “Abnormal” if a ground-truth anomaly was located within its or its neighbor’s boundaries, and “Normal” otherwise.

The standard deviation of the vibration signal was chosen as the primary analysis metric. In the field of condition monitoring, the standard deviation is a well-established time-domain feature that directly quantifies the energy of a vibration signal; an increase in this value is a classic indicator of a system responding to a fault or anomaly. The standard deviation was calculated for each segment and then averaged across all runs for a given channel to produce a robust profile of the track’s condition, as shown in Fig. 7.

Fig. 7.

Track vibration analysis by segment. The bar chart shows the standard deviation of the z-axis acceleration for each 7.62-m track segment, averaged across all relevant runs. Segments containing known ground-truth anomalies are highlighted in red.

The results indicate a consistent, though modest, increase in the average standard deviation for abnormal segments across all six sensor channels, providing preliminary evidence of anomaly-related signatures in the data. Interestingly, the analysis also highlights the effect of sensor location on signal. The sensors mounted on the bogie frame (Channel 1 & Channel 3) show the most distinct separation between normal and abnormal segment values. This is physically intuitive; the axle-mounted sensors (Channel 2 & Channel 4) are subject to a high degree of noise from the direct wheel-rail interaction, which can obscure the anomaly signatures, while the sensors on the car body (Channel 5 & Channel 6) measure a signal that has been significantly damped by two stages of suspension. This behavior is consistent with the physical nature of the dataset: the naturally occurring anomalies are relatively subtle irregularities embedded within a complex, multi-source vibration environment, and the standard deviation is computed over 7.62-m segments that integrate contributions from both anomalous and surrounding normal track. It is emphasized that the purpose of this analysis is to validate that anomaly signatures are present in the data, not to optimize detection performance.

Visual Data Validation

A series of checks was performed to validate the quality, integrity, and utility of the vision data for computer vision-based analysis. Figure 8 shows exemplary frames captured at a few highlighted locations.

Fig. 8.

Example frames from the vision system. The central map shows the location along the test track where each corresponding image was captured, illustrating the clear view of the rail and fastener components under different conditions.

First, a direct manual review of the video streams was conducted to confirm that the physical reference markers (cones) used for spatial calibration are visible at their expected locations in the video, providing direct validation of the ground-truthing and localization procedures. Second, the temporal and spatial continuity of the visual record was validated. The effective frame rate of 30-40 FPS, even at the highest targeted operational speed of 27.5 km/hr, ensures a significant spatial overlap between consecutive frames. This provides a continuous, unbroken visual record of the entire test segment, which is critical for applications such as object tracking or change detection, ensuring no gaps in coverage.

Then, to quantitatively assess the quality of the imagery, the sharpness of each frame was evaluated using the variance of the Laplacian operator. This is a well-established metric in computer vision where higher values correspond to sharper images with more defined edges, while lower values indicate blur. The analysis revealed a consistently high level of sharpness across the entire dataset, with an average Laplacian variance of 651.77. To contextualize this value, the same metric was computed for images collected while the train was stationary, representing the upper bound of sharpness achievable under the same camera configuration and environmental conditions; these static images yielded an average Laplacian variance of 964.85. These two values confirm that the global shutter camera was effective at mitigating motion blur and that the dataset consists of focused images suitable for detailed algorithmic analysis.

Finally, to further validate the vision data for anomaly detection beyond confirming its sharpness and continuity, a self-supervised learning study was conducted using the captured image. The core principle of this approach is to train a model exclusively on imagery of normal track conditions, enabling it to learn a robust representation of the healthy track state. During inference, track sections that deviate from this learned norm produce high reconstruction errors, which serve as anomaly indicators without requiring any labeled defect data.

The pipeline operates as follows. Each image is first preprocessed by applying a mask that isolates the region of interest, encompassing the rails, fasteners, ties, and ballast bed, while excluding irrelevant background elements such as vegetation. A reconstruction-based deep learning model is then trained on normal track imagery. Four model configurations were benchmarked, formed by combining two reconstruction architectures, a deterministic autoencoder (AE) and a probabilistic variational autoencoder (VAE), with two feature extraction backbones, a convolutional neural network (CNN) and a Vision Transformer (ViT). During inference, the pixel-wise reconstruction error for each image is collected.

The image-level anomaly detection performance of each model was quantified using the area under the receiver operating characteristic curve (AUC), with scores ranging from 0.65 to 0.73 across the four configurations, as shown in Table 3. Across all models, a consistent and meaningful separation between the reconstruction errors of nominal and anomalous images was observed.

Table 3.

Summary of the anomaly detection performance for the four benchmarked models. Trainable Parameters indicate model size. GFLOPs measure the computational cost for a single forward pass. Training Time reflects the duration per epoch on our hardware at a batch size of 32 and a training set of 40,000 instances, running on A100-HighRAM from Google Colab. The AUC Score quantifies the model’s ability to distinguish anomalous images from normal ones based on reconstruction error.

Model Combination	Trainable Parameters (M)	GFLOPs	AUC Score
CNN AE	19.03	0.6	0.65
CNN VAE	19.16	0.6	0.68
ViT AE	15.08	3.5	0.72
ViT VAE	15.21	3.5	0.73

Author contributions

Sizhe Ma contributed to the conceptualization, methodology, software, validation, formal analysis, investigation, data curation, writing - original draft, writing - review and editing, and visualization. Katherine A. Flanigan and Mario Bergés contributed to the conceptualization, methodology, validation, investigation, resources, writing - review and editing, supervision, project administration, and funding acquisition. James D. Brooks contributed to the conceptualization, methodology, validation, investigation, writing - review and editing, and supervision.

Data availability

The full dataset, including the processed, synchronized time-series data and the corresponding image frames for each run, is available in the Hugging Face repository at https://doi.org/10.57967/hf/8411⁴².

Code availability

The custom Python scripts used for all computational processing and analysis described in this manuscript are provided in the Hugging Face repository at https://doi.org/10.57967/hf/8411⁴².

Competing interests

The authors declare no competing interests. Mario Bergés holds concurrent appointments at Carnegie Mellon University (CMU) and as an Amazon Scholar. This manuscript describes work at CMU and is not associated with Amazon.