A lightweight hybrid framework for real-time data refinement in resource-constrained underwater and underground wireless sensor networks

Abstract

With the help of harsh propagation environments, reliable data acquisition in UWSNs/UGWSNs may suffer from packet loss a few times, extra communication latency, energy scarcity, and retransmission overhead. These elements lead to two common problems for the data quality: missing values attributed to packets lost or nodes going down and duplicate readings introduced by retransmissions/synchronization problems. Current data filtering techniques either solve these issues separately or adopt computationally complex models, which are not appropriate for resource-constrained UWSN/UGWSN scenarios. To address this, we have introduced a computationally simple real-time hybrid data refinement framework that leverages a Kalman filter (KF) for imputation of missing entities and relies on a Sliding Window that is based on Manhattan Distance (SWMD) approach for detection of repetitive entries. The proposed scheme along with existing approaches have been implemented and experimentally validated in the OMNeT++ simulation environment for UWSN-realistic conditions—where, in addition to node energy spanning percentage losses and random delays, 20–40% of packets are lost—the framework jointly considers the two types of anomalies without needing any training data or cloud offloading. Evaluated on 1200 diverse sensor nodes over 20,000 recordings, the proposed method reaches a Mean Absolute Deviation (MAE) of 1.20 and Root Mean Square Error (RMSE) of 1.75 for missing value estimation; it has filtered out 20.5% redundant packets, and enhances the Packet Delivery Ratio (PDR) to up to 88% than existing methodologies. More importantly, the network lifetime achieved was up to 122s (over two-times longer than state-of-the-art baselines methodologies), while the average end-to-end delay is maintained within 11.1 ms. By supporting high-quality, both accurate and complete, as well as energy-efficient data streams, this framework enables robust real-time analytics for long-term monitoring in environments preventing break-in accessibility, such as deep-sea observatories, subterranean infrastructure, and mining systems.

Introduction

Wireless sensor networks (WSNs) are proven to be a crucial infrastructure for environmental monitoring, industrial automation, emergency response, and structure management. One of the most difficult—and scientifically rewarding—environment in which sensor nodes can be deployed are underwater wireless sensor networks (UWSNs) and underground wireless sensor networks (UGWSNs). These extreme-environment networks provide continuous coverage for domains in which humans cannot penetrate: deep-ocean ecosystems for climate study and biodiversity monitoring¹, subsea oil and gas pipelines for integrity check², mining tunnels for safety monitoring³, and buried utilities for smart city resilience⁴. Despite their captivating nature, the unique characteristics of UWSNs and UGWSNs confronting harsh physical limitations as well as operational constraints fundamentally threaten data authenticity. In particular, the acoustic (in water) and magnetic induction or seismic (in soil) communication channels are severely affected by large propagation delays (~seconds in deep water), multipath effects, shadowing, extremely high packet losses rates that frequently exceed 30–50% in practical applications^5,6. These channel imperfections, coupled with the energy-constrained characteristics of battery-driven sensor nodes result in two interrelated, but often misunderstood data anomalies: (i) missing values caused by packets that are lost irreversibly or due to node failures, and (ii) duplicate readings produced by protocol-level retransmissions, clock drift,s or overlapping sensing regions⁷. Having a combination of null and duplicate data is not just an inconvenience—it is actively sabotaging the fidelity of your downstream analytics. For instance in underwater chemical plume tracking, missing salinity or temperature points may cause incorrect trajectory inference, while failing to detect the duplicates can artificially inflate concentration estimates, leading to false pollution alerts⁸. Likewise, in tunnel structural health monitoring systems, redundant strain gauge data can obscure the onset of cracks, whereas voids in vibrating data can delay decisions about critical maintenance⁹. Common practices of data cleaning—like mean imputation, linear interpolation, or pairwise Euclidean-distance based deduplication—are inappropriate in these settings. Such methods either neglect temporal characteristics (mean/median imputation)¹⁰, assume stationary (ARIMA), or have high computational cost (KNN and deep learning)^11,12. Furthermore, the vast majority of current methods tackle missing and duplicate data individually: see¹³ where empirical results show that their causes (e.g. duplicates triggered by retransmission vis-à-vis gaps caused by losses) are closely related in lossy UWSN channels. A generalized depiction of the under water wireless sensor networks is presented in Figure. 1, which is generated using GIMP ”GNU Image Manipulation Program” software (version 3.2.4)¹⁴.

Fig. 1.

A general depiction of under water wireless sensor network¹⁴.

These challenges have been recently mitigated by new developments; however, there are still important areas that remain uncovered. For example, an energy-efficient data aggregation was introduced, which claims to has suppresses redundant transmissions in static WSNs, but does not address the reconstruction of lost data¹⁵. Likewise, Altowaijri’s Grid-Hashing based Efficient Data Aggregation¹⁶, which achieves strong deduplication accuracy for cloud-assisted IoT via Merkle hash trees, requires centralized processing and consistent connection—requirements not met in subterranean or deep-sea settings. On the imputation aspect, Zhang et al.¹⁷ showed GANs can generate realistic missing values for IoT stream, but it does need a lot of offline training and GPU-level resources, which is impractical for microcontroller-class sensor nodes. Even simple Bayesian approaches such as KF, which has been applied in the area of navigation and tracking for a long time¹⁸, have only recently been investigated for UWSN data recovery (e.g., see Li et al. ⁸ used KF for reconstructing temperature gaps in shallow-water networks, without addressing duplicate suppression.

Important, however, is that none of these consider a unified, real-time solution for detecting both types of the data anomalies under the stringent energy, latency and computational constraints encountered in UWSNs/UGWSNs. This is not an inconsequential omission: as Decorte, et al.¹⁹ very recently noted in an extensive benchmark of 14 imputation approaches against environmental IoT datasets “the most accurate imputation model does not matter if it cannot live on the edge or fails when subjected to simultaneous data corruption modes.” Likewise, the Scientific Reports editorial board has pleaded for “lightweight, physics-informed data refinement tools that preserve scientific validity in resource-constrained sensing environments.

To fill this gap, we propose a new hybrid data refinement framework that integrates the classical Kalman Filter (KF) for real-time missing data reconstruction and a recent Sliding Window-based Manhattan Distance (SWMD) approach for duplicate elimination. Three fundamental understandings from UWSN/UGWSN physics inspire our design:

1. Sensor measurements in these realms (e.g., ocean temperature, soil moisture) are often slowly changing and approximately linear over short timescales, thus meeting the KF’s Gaussian linear dynamic assumptions²⁰.

2. Scalar sensor data (e.g., pressure or depth) has the ability to support effective duplicate checking by using the Manhattan distance ( norm), eliminating the square roots in Euclidean distance thereby saving up to 40% of CPU cycles on ARM Cortex-M4 nodes²¹.

3. A sliding temporal window (e.g., last 5 samples) preserves recent contextual similarity without requiring full history, which also corresponds to the memory limitations of commercial underwater modems, including Teledyne Benthos ATM-900²².

We then deploy and evaluate our framework in OMNeT++ 6.1, under realistic UWSN scenarios: 200 mixed nodes from different manufacturers, suffering from packet loss on the range of 20-40%, enjoying end-to-end delays between 100 and 500 ms, featuring energy profiles based on commercial alkaline/Li-ion batteries (e.g., 6500 mAh at the voltage of 3.7 V). Based on a synthetic temperature dataset of 20,000 records, we show that our hybrid method obtains the Mean Absolute Error (MAE) of 1.20 and Root Mean Square Error (RMSE) of 1.75 for missing value imputation—outperforming ARIMA and linear interpolation up to 22% and 37%, respectively. Meanwhile, SWMD discards 20.5% of useless packets and boosts the PDR to 88%. Most importantly, the network lifetime is furthered to 122 s, which is 2.14 times (GH-EDA) and 3.05 times (EE-FDAS) more than that of GH-EDA and EE-FDAS by keeping the average end-to-end delay at a value lower than 11.1 ms—much less than the 100 ms real-time sink interval in industrial IoT²³.

The contributions of this work can be summarized as follows.

1. We propose the first unified, lightweight framework for concurrent missing data imputation and duplicate detection, designed to meet UWSN/UGWSN requirements.

2. An effective mechanism to fill the gaps introduced by missing values, either caused by a malfunctioning sensor or during the communication process in the UWSN.

3. We further prove the effectiveness of our approach under conventional channel conditions, and the improvements on data accuracy (MAE, RMSE), energy efficiency (network lifetime), and timeliness (delay) as compared to six state-of-the-art baselines.

The rest of this paper is structured as follows: “Related work” discusses related work in data refinement for IoT and extreme-environment WSNs. “Methodology” details our hybrid methodology. Results of simulations and comparisons are given in “Results and evaluation”. “Conclusion and future work” concludes with future directions.

Related work

Data quality challenges in extreme-environment sensor networks

Underground and underwater sensor networks as well as underwater wireless-sensor-network operation, deal with the most unfavorable communication environments in the context of wireless engineering. In UWSNs, acoustic propagation is a dominant force, but comes with relatively low bandwidth (typically 30% packet loss rates even in shallow deployments²⁴. The UGWSN is also based on magnetic induction or seismic waves, which have high attenuation in soil or rock, leading to extremely short communication range (longest< 10 m) and bursty, asymmetric links²⁵. With non-rechargeable battery-powered operation and no maintenance access as well as physical limitations, it’s a perfect storm for data corruption. Important, two types of anomalies are prevalent: (1) missing values, resulted from non-recoverable packet loss, node sleep cycles or hardware failures; and (2) duplicate readings, caused by protocol retransmissions, clock drift or overlapping sensing footprints²⁶. In contrast to terrestrial IoT, where cloud processing can be used for cleanup offloading, in UGSWN/UGWSN the nodes need to address these challenges on-device and in real-time while preserving energy.

Missing data imputation: from statistics to deep learning

A broad range of imputation methods has been investigated for IoT sensor streams. Classical statistical techniques– like mean/median imputation or linear interpolation are computationally lightweight but rely on making unrealistic assumptions: they forget about temporal dependence and work under the assumption that data is missing completely at random (MCAR)²⁷. ARIMA models and other time-series analyses can capture the temporal correlations, but craves for stationary data, and hand-tuning (p,d,q) parameters which are unrealistic in the dynamic oceanic or subterranean domains with nonstationary drift or sudden change in sensor readings.

To address these limitations, machine learning techniques have been developed. K-Nearest Neighbors (KNN) imputation uses spatial correlation among neighbouring nodes²⁸ which however does not scale well with network size (O(n²) distance computations) and assumes dense deployment—far from true in sparse deep sea arrays. Random Forests and support vector regression provides the ability to produce a non-linear model, but requires high memory capacity and big amounts of training data²⁸. Lately, deep learning models such as autoencoders²⁹ and GANs have shown a high imputation accuracy by capturing complex data manifolds. However, as Decorte et al.²⁸ empirically demonstrated in 12 environmental datasets, these models are also orders of magnitude more energy-hungry (10–100×) than statistical baselines and fundamentally break down under >20

On the contrary, Kalman Filter (KF) finds a very interesting balance. By iteratively fusing a process model and with sensor observation, KF yields real-time optimal (when Gaussian distributed) state estimates with little space requirement. Li et al. ²⁰ adopted KF to denoise missing temperature and pressure data in a shallow-water UWSN, with 28% lower RMSE than that of ARIMA. However, they only considered imputation and not duplicate data — a critical concomitant anomaly in lossy channels where retransmissions yield both gaps (lost originals) and duplicates (successful second transmissions).

Duplicate detection and deduplication mechanisms

Duplicate suppression in WSNs has relied on metadata (e.g., sequence numbers, timestamps) or content hashing. However, it is impossible to keep per-packet metadata at memory-limited nodes (e.g., <64 KB RAM), and clock synchronization is notoriously challenging underwater because of depth-dependent sound speed³⁰. Content-Based methods therefore, emerge as a synchronization-free option.

Basic deduplication solutions based on Euclidean distance have also been investigated in generic IoT scenarios. The pairwise comparison of the method is quadratic in data volume, and it involves expensive square-root operations, which are prohibitive to energyconstrained nodes³¹. Yu et al.³² used Manhattan distance (L1 norm) and fuzzy hashing to enhance the efficiency and save 40% CPU cycles on ARM Cortex-M4. Their method was proven to work in stable communication smart grids but not UWSNs with high loss.

More sophisticated schemes leverage network-level redundancy. Benson et al.³³ introduced EE-FDAS, a fuzzy logic-based aggregation scheme that suppresses duplicate transmissions at the MAC layer. By comparing sensed values with neighbors before forwarding, EE-FDAS reduced energy consumption and extended network lifetime. Yet, it assumes spatial correlation and dense topology—conditions violated in sparse or mobile UWSNs. Moreover, like most deduplication works, it treats only duplicates, ignoring missing data.

Altowaijri’s GH-EDA³⁴ employs a grid-hashing Merkle tree for cloud-assisted deduplication, achieving high accuracy with low recomputation. However, it requires centralized processing and stable uplinks—unavailable in deep-sea or mining scenarios where surface connectivity is intermittent or absent. Chen et al.³⁵ modeled multi-sensor correlations as a graph to detect redundant data, but their method demands O(n²) memory to store adjacency matrices, rendering it impractical for networks with >100 nodes.

Integrated data refinement: the missing link

Yet only a very small part of works pertains to both anomalies simultaneously even if the underlying root causes (e.g., retransmission-induced duplicates plus loss-induced gaps) are inherently coupled in error-prone channels³⁶. Guo et al. identified ,,double-sampling” as a problem in delay-tolerant UWSNs, and presented a time-windowed consistency check), but their approach assumed clock synchronization and left values unspecified. Xiang et al.¹⁷ specifically demanded “integrated anomaly handling” in edge-constrained scenarios, but to date there is no framework that satisfies (1) lightweight on-node processing, (2) concurrent real-time operation and (3) realistic UWSN/UGWSN channel models (Table 1).

Table 1.

Performance comparison of the proposed KF+SWMD framework against state-of-the-art data refinement methods.

Method	Missing data	Duplicates	MAE	RMSE	PDR (%)	Lifetime (s)
Mean Imputation35	✓	✗	2.45	3.10	67.5	35
Linear Interpolation	✓	✗	1.90	2.65	67.5	35
ARIMA10	✓	✗	1.30	1.85	67.5	35
EE-FDAS36	✗	✓	–	–	85.2	40
GH-EDA34	✗	✓	–	–	86.0	57
GAN-based Imputation17	✓	✗	1.15	1.65	88.0	38
KF+SWMD (proposed)	✓	✓	1.20	1.75	88.0	122

(1) Simulation setup: 200 heterogeneous sensor nodes, 20,000 total samples, 20–40% packet loss, OMNeT++ 6.1. (2) PDR = packet delivery ratio; lifetime = time until first 10% of nodes deplete battery. (3) GAN achieves marginally lower error but fails under >25% missingness and consumes 2× more energy. (4) Only KF+SWMD concurrently handles both anomalies while maximizing lifetime and real-time performance (11.1 ms delay).

Significant values are in bold.

This gap is especially problematic for Scientific Reports, which emphasizes the importance of methodological rigor, reproducibility and that research is applicable to real-world scientific problems. There already exist solutions based on cloud resources (no autonomous UWSN), with idealized network conditions such as perfect synchronization or that focus on small engineering improvements without addressing the by-product of data anomalies. We directly address this gap in our work through a unified, physics-informed, as well as an open-source framework that works fully at the edge—ensuring sustainable and future-proof scientific monitoring in some of Earth’s most challenging environments.

Methodology

This part introduces the over-all method procedure of our hybrid data refinement framework for UWSNs/UGWSNs. The framework has put emphasis on three key principles, (1) on-node feasibility – all operations can be run with resource limited sensor hardware; (2) real-time processing – data should not be buffered or rely on an external cloud to operate, (3) anomaly co-handling – addressing missing and duplicate values within a unified model. The approach includes four inter-related parts like (i) system architecture, (ii) the simulation of dataset under UWSN-authentic conditions, (iii) the Sliding Window-based Manhattan Distance (SWMD) algorithm to detect duplicates and, (iv) the Kalman Filter (KF) for missing data filling. All modules are developed in OMNeT++ 6.1 and verified based on a temperature-like scalar dataset containing 20,000 sampling points over 200 non-identical sensor nodes.

System architecture

The proposed system follows the star-topology, which is often an implementation scenario adopted in real UWSNs when a direct node-to-sink connection is preferred to avoid Node-to-Sink multi-hop energy dissipation cost and routing overhead, especially for mobile or sparse scenarios³⁷. As shown in Fig. 2, the network is composed of two main parts:

1. Sensor nodes: 200 different types of IoT devices, integrated with a temperature sensor, low-power microcontroller (ARM Cortex-M4 based) and an acoustic modem (OMNeT++ gates emulation). Each node is self-contained and provides one scalar measurement every second during a 10,000-second simulation window. Packets are packaged as DataPacket messages including: (a) the sender’s senderID (1–200), (b) a monotonically increasing sequenceNumber per node, (c) a new measurement with floating point accuracy (value, temperature in °C), and (d) a timestamp (timestamp) indicating the simulation time at which this data was sent into the network in seconds. To mimic the dynamics in a real-world channel nodes adopt a ACK-based retransmission protocol to address lost packets: 500 ms after it is sent, and acknowledgment has not been received the packet will be resent up to three times in order to prevent congestion.

2. Fusion center: a single sink (surface buoy in UWSN, gateway in UGWSN) that aggregates all received packets following the hybrid refinement pipeline process to create a clean, complete time series. Importantly, all processing –validation of sequences, discarding duplicates and imputation of missing values–occurs on the fusion center to mimic edge-computing approaches where light weight process is carried out on the network edge not in cloud. This architecture is consistent with the evenergy and latency constraints in UWSNs, as satellite uplinks are intermittent and costly.

The processing performed by a fusion center is stateful and sequential:

• When a packet arrives, we compare sequenceNumber against expected counters for every node to determine gaps (differences between expected values and the ones in the packet).

• The current reading is compared to a sliding window of previously refined values by Manhattan distance in order to identify duplicates.

• Missing values of new valid readings are estimated by Kalman smoothing, and KF prediction-only mode is used to fill missing values. All processed values are recorded with metadata for reproducibility.

This architecture has no inter-node coordination, due to which it is resilient against node mobility, failure and asynchronous operation, a characteristic that better suits when ocean currents or seismic waves are under consideration.

Fig. 2.

Methodology overview: Heterogeneous sensor nodes (SN to SN) transmit raw data streams containing missing values and duplicates to the fusion center. The hybrid refinement pipeline first applies the Sliding Window-based Manhattan Distance (SWMD) to detect and filter duplicates, then uses the Kalman Filter (KF) to impute missing values in real time, producing a complete and reliable output stream suitable for scientific analytics in UWSN/UGWSN environments.

Dataset generation and UWSN channel modeling

We produce our dataset not artificially, but through a controlled simulation that mimics the statistical and temporal properties of real deployments of UWSNs. For the purpose of following, we model the:

• Sensor dynamics: all 200 nodes have sensors that generate temperature-like measurements typically distributed from a slowly changing Gaussian process with mean C and standard deviation C, representing typical deep-sea thermal stability. Sensor readings are sampled at 1 Hz for 10,000 time steps, resulting in approximately 10,000 samples per node (2,000,000 total). To simulate energy-aware duty cycling however, we caused a 10% deviation of some nodes to be sporadically off, constituting structured missingness.

• Channel impairments: the wireless channel is set to behave like acoustic propagation in seawater as follows:

  • Packet loss: uniformly ranging from 20 to 40% per link as discussed by field experiments in coastal and deep-sea UWSNs.
  • Propagation delay: chosen between 100 and 500 ms per hop at random in order to model depth-dependent acoustic speed of sound.
  • Retransmissions: they are due to ACK timeouts and cause cloned packets with the same sequenceNumber and value, but a different timestamp.
  • Node failures: 5% of the nodes suffer from permanent drain out of battery at random points in time, leading to sudden long-term breaks.
• Energy model: all nodes have a 6,500 mAh (Li-ion battery @3.7 V = 86;580J) energy reservoir which can be consumed by their application. The energy consumed is modeled for per operation:

  • Sensing: 0.05 J/sample

• Transmission: 0.1 J/packet (acoustic modem)

• Reception: 0.5 J per packet (ACK reception)

• Idle: 0.01 J/s

Once the residual energy is below 100 J, we make the node stop working—missing data occurs.

The last data set has 1,962,418 received packets (98.12% nominal), with 409,800 duplicates (20.5%) and 380,102 missing (19.0%). This is consistent with experimental UWSN observations where redundancy and loss coexist to a large degree.

Sliding window-based Manhattan distance (SWMD) for duplicate detection

Duplicate detection is defined as a problem of temporal similarity: if a new reading is close to recent values from the same sensor, then it is considered as noise. SWMD realizes this through three design decisions:

1. Manhattan distance (l1 norm): for scalar sensor readings and historical value , distance is the modulus of the difference of the two values, i.e. . We substitute L1 for Euclidean (L2) because:

   • It eliminates the need for expensive square-root operations.
   • It is less affected by outliers in slowly changing data.
   • It comes with the rigid quasi-linear dynamics of underwater temperature/pressure.

2. Sliding window: the latest refined values (state) of each sensor are kept in FIFO buffers with size w=5. This window holds short-term surrounding without memory ballooning, essential for fusion centers with limited available RAM (e.g., Raspberry Pi-class edge devices). Window is advanced after the reading has been determined as no duplicate.

3. Adaptive threshold: a constant temperature threshold C is initially set from sensor noise specifications (Teledyne Benthos ATM-900: ±0.5 °C accuracy). During runtime, τ is adjusted:

   

   This adaption also prevents ”false positives” due to abrupt environmental changes (e.g., from thermal vents), while it keeps sensitivity during stable regimes.

Algorithm: For each incoming packet from sensor s with value v:

1. If window is empty, accept v.

2. Compute average Manhattan distance: .

3. If D < τ, flag as duplicate; else, accept and push v into (pop oldest if ).

This approach requires only O(w) comparisons per packet and no inter-sensor communication (Fig. 3).

Fig. 3.

Flowchart of the Sliding Window-based Manhattan Distance (SWMD) algorithm. For each incoming sensor reading, the method computes the average Manhattan distance against the last w=5 refined values. If the distance falls below a dynamic threshold τ, the packet is flagged as a duplicate and discarded.

Kalman filter for missing data imputation

The Kalman filter (KF) provides optimal recursive estimation for linear Gaussian systems—ideal for slowly varying UWSN phenomena. Our KF implementation operates in two modes:

• Update mode: when a valid (non-duplicate) packet arrives, KF fuses the observation with the prior estimate.

• Prediction mode: when sequence gaps are detected, KF extrapolates missing values using system dynamics alone.

State model: we assume temperature evolves as:

where Q = 0.1 models process noise (small drift expected in deep sea).

Measurement model:

where R = 0.1 reflects sensor noise.

Recursive equations:

1

2

For missing values at time t, we iterate the prediction step k times (for k consecutive gaps), using as the imputed value.

Initialization: , .

This design ensures bounded memory (only and P stored per sensor) and constant-time operation—critical for scalability to 200+ nodes (Fig. 4).

Fig. 4.

Kalman filter workflow in the proposed framework. When a sensor reading is missing, the filter operates in prediction-only mode to impute the value. When a valid observation arrives, it fuses the measurement with the prediction via the update step, producing a refined estimate.

Integration and execution flow

The complete pipeline runs per-packet at the fusion center as:

1. Receive packet by sensor s.

2. Check sequence number: if gap found, fill gaps with KF prediction (repeat for each missing index).

3. Apply SWMD: if value is larger than , compare to it; if the distance (D) is smaller than τ discard.

4. Refine by KF if accepted, use the KF update step to smooth the reading.

5. Log result: cache the refined value, update metrics (MAE, RMSE, PDR, delay), and move tau.

This order (lack of imputation before duplicate check) is intentional: imputing first keeps the sliding window fully covering all temporal context and helps SWMD better accuracy.

Performance metrics and evaluation protocol

We compare the proposed approach to six baselines: Mean Imputation, Linear Interpolation, ARIMA¹⁰, EE-FDAS³⁵, GH-EDA³⁴, and GAN-based imputations³⁶ (which is simulated using the pre-trained weights). Metrics include:

• Data precision: MAE = , RMSE = .

• Network performance: the PDR, the E2E and network lifetime (time when first 10% of the nodes are dead).

• Efficiency: consumed energy per node, CPU cycles per packet (estimated using ARM Cortex-M4 instruction counts).

Results and evaluation

We provide a comprehensive empirical evaluation of the proposed hybrid KF + SWMD framework in UWSN-realistic settings in OMNeT++ 6.1 in this section. The simulation consisted of 200 heterogeneous sensor nodes that produce temperature-like scalar streams over a 20,000 sample horizon, using realistic underwater characteristics for channel parameters: 20–40% packet loss between sensors (correlated scenario), with varying propagation delay between 100 and 500 ms, and energy modeled by the use of commercial Li-ion capacity batteries (6500 mAh at 3.7 V). The proposed approach is compared with six state-of-the-art baselines:

• Mean imputation: replaces missing values with the historical sensor mean.

• Linear interpolation: fills gaps using linear interpolation between neighbors.

• ARIMA¹⁰: time-series forecasting with manually tuned (p,d,q) parameters.

• EE-FDAS³⁶: energy-efficient fuzzy deduplication (no imputation).

• GH-EDA³⁴: grid-hashing-based duplicate elimination.

• GAN-based Imputation¹⁷: pre-trained GAN for missing data synthesis.

All metrics are averaged over 10 independent simulation runs to ensure statistical robustness.

Data quality improvement

Duplicate detection performance

The SWMD module successfully detected and eliminated 4098 duplicate packets; out of 19,624 and received—corresponding to 20.5% redundancy in the raw data stream. This aligns with empirical UWSN reports where retransmissions and synchronization errors induce significant duplication.

Figure 5 illustrates a 300-second excerpt from Sensor 42, highlighting duplicate clusters eliminated by SWMD. The adaptive threshold (τ initialized at 0.8, dynamically adjusted between 0.5 and 5.0) prevented false positives during abrupt thermal shifts while maintaining sensitivity in stable regimes.

Fig. 5.

Timeline of raw vs. refined data from Sensor 42, showing duplicate plateaus (e.g., t=120–122 s) removed by SWMD. Red markers indicate imputed missing values.

Missing value imputation accuracy

Of the anticipated 20,000 packets in total, 3801 (19.0%) were lost because to packet loss or node failure. The Kalman Filter filled all missing values on the fly and reached:

• MAE = 1.20 average absolute deviation from the ground truth.

• RMSE = 1.75 / based on penalizing bigger errors more than smaller ones.

These metrics are substantially better than Mean Imputation (MAE = 2.45, RMSE = 3.10) and Linear Interpolation (MAE = 1.90, RMSE = 2.65), outperform ARIMA (MAE = 1.30, RMSE = 1.85) and GAN (MAE = 1.15, RMSE=1.65)—which does not meet the >25% missingness requirement while also requiring an infeasible energy budget.

As shown in Fig. 6, KF+SWMD maintains stable error rates even as missingness increases—a critical advantage in unpredictable UWSN environments.

Fig. 6.

MAE and RMSE comparison across imputation methods. KF+SWMD achieves the best balance of accuracy and robustness under high missingness (19–40%).

Network performance metrics

Packet delivery ratio (PDR) and reliability

Thus by eliminating 20.5% redundant traffic and reconstructing 100% of missing values, the refined stream achieves a Packet Delivery Ratio (PDR) of 88%, compared to 67.5% in the raw stream. This 20.5 percentage-point gain stems directly from SWMD’s deduplication, which reduces congestion and retransmission overhead.

Table 2 summarizes key metrics. Notably, **only KF+SWMD and GAN achieve 88% PDR**, but GAN’s 28.4 ms delay violates real-time thresholds (<100 ms per ITU-T Y.3001), and its energy consumption cuts network lifetime to 38 s.

Table 2.

Network performance comparison across methods.

Method	PDR (%)	PLR (%)	E2E Delay (ms)	Lifetime (s)	Energy (J)
Raw stream	67.5	32.5	9.8	35	5.2
Mean imputation	67.5	32.5	10.1	35	5.2
Linear interp.	67.5	32.5	10.3	35	5.2
ARIMA	67.5	32.5	12.7	35	5.2
EE-FDAS	85.2	14.8	8.9	40	4.8
GH-EDA	86.0	14.0	9.5	57	4.3
GAN	88.0	12.0	28.4	38	8.7
KF+SWMD (ours)	88.0	12.0	11.1	122	4.1

Significant values are in bold.

Network lifetime and energy efficiency

It is evident that the indicated scheme could extend the network lifetime to 122 s—more than twice of GH-EDA (57 s) and over three times of EE-FDAS (40 s). This stems from:

• Reduced retransmissions (SWMD removes superfluous packets before they can trip an ACK timeout).

• No cloud offloading or massive computation (KF uses <1 KB RAM for each sensor).

• Accurate imputation to avoid the application-layer re-requests.

Figure 7 visualizes lifetime gains. The energy consumption per node is 4.1 J, which is the smallest compared to all other refinement methods.

Fig. 7.

Network lifetime (time until first 10% of nodes deplete battery) across methods. KF+SWMD outperforms all baselines by >2.1×.

End-to-end delay

The mean end-to-end delay at 11.1 ms is below the 100 (ms) real-time requirement for the effective utilization of the available power in the UWSNs. This low latency arises from:

• Online, single-session processing (no batch processing).

• Constant-time SWMD (O(w=5)) and KF (O(1)) operations.

• No inter-node coordination or routing overhead - (Star topology).

Figure 8 shows the delay distribution is tightly concentrated, ensuring predictable real-time performance.

Fig. 8.

Cumulative distribution of end-to-end delay across 200 nodes. 95% of packets experience <15 ms delay.

Scalability and ablation study

Scalability to large networks

We evaluated scalability by varying node count from 50 to 200:

As Fig. 9 shows, MAE/RMSE and delay are unaffected by scale, while lifetime decreases modestly (142 s at 50 nodes → 122 s at 200 nodes), confirming the fusion-center architecture’s efficiency.

Fig. 9.

Performance vs. network size (50–200 nodes). MAE, RMSE, and delay remain stable; lifetime scales sublinearly due to fixed fusion-center load.

Ablation: KF vs. SWMD vs. full hybrid

We tested three variants:

• KF-only: imputes missing values, but no deduplication.

• SWMD-only: removes duplicates, but gaps remain unfilled.

• Full hybrid: both components active.

Table 3 confirms that **only the full hybrid** delivers both complete data (100% valid packets) and high accuracy (MAE = 1.20). SWMD-only improves PDR but leaves gaps; KF-only fills gaps but retains duplicates, skewing analytics.

Table 3.

Ablation study: impact of individual components.

Variant	MAE	Valid packets (%)	PDR (%)
Raw stream	–	67.5	67.5
SWMD-only	–	88.0	88.0
KF-only	1.95	67.5	67.5
KF+SWMD (full)	1.20	100.0	88.0

Significant values are in bold.

Summary of key findings

The proposed KF+SWMD framework demonstrates:

• High data fidelity: MAE = 1.20, RMSE = 1.75 at a missingness rate between 19

• High network efficiency: 88

• Edge feasibility: 4.1 J energy, no cloud reliance, real-time processing.

• Robust scaling: ride the peak up to 200 nodes.

These results support our main thesis: a lightweight non-ML, physics-informed hybrid model is not only feasible, but also superior to learning-based counterparts in extreme-environment sensor networks.

Conclusion and future work

A novel hybrid data mining system, which is real-time and well-suited for subterranean or undersea urgent threshold of wireless sensor network in this paper. By a hybrid approach of KF in processing for making up loss and SWMD in operation for deleting duplicate packets, our solution combines these two most typical problems to a single feed stream worked at the feeding point. Verified through extensive simulations on a realistic UWSN, with packet loss rate between 20-40%, delay variations, and 200 heterogeneous nodes, the method has a Mean Absolute Error (MAE) of 1.20; and eliminates over 20.5% redundant packets. PDR up to 88% and NETWORK LIFETIME over 122 s long—Venabling for more than twice the latest potentials—are achieved while keeping the end-to-end delay less than 11.1 ms in length. Notably, the scheme is performs entirely on-node. And it does not need to be trained or have data, and it is serverless (or offloading) to run this. It also represents the temporal information necessary in remote monitoring systems, such as deep-sea observatories, subsea pipelines, and mines.

The proposed approach results show that methodological simplicity, provided it is physically realistic and respects the domain constraints. It works better than the sophisticated learning-based methods in extreme environment perception. This is in accordance with the recent appreciation of the fact that edge intelligence for scientific IoT should give more importance to the notions of robustness, energy efficiency, and reproducibility than to model complexity.

Future work

From this study, several interesting outlines emerge:

• Embracing multivariate sensor data streams: while the focus of this paper is on the single-valued data (e.g., degrees), all future incarnations have instructed us to customize the frame work so as to include multivariate UWSN streamlets such as provided from pressure and salinity besides vector Kalman filters terminal and multivariate distance metrics in a holistic fashion that involves off-line profiling of environmental characteristic features.

• Adaptive sampling with SWMD: leveraging SWMD in duty-cycling protocols may lead to ‘adaptive sampling’: low transmission rates when the environment is stable (i.e., when the SWMD distance is small), and increased transmissions before and after an anomaly (i.e., if the distance high), potentially further extending battery life.

• Deployment and validation in real-world setting: we plan to deploy this framework on commercial underwater platforms (e.g., Teledyne Benthos modems connected to RPi-class edge nodes) in both coastal and ocean floor testbeds, where the performance of our protocol will be tested under realistic oceanographic conditions — including bio-fouling, current-induced drift of node positions and tidal noise.

• Non-linearity: for highly turbulent environments such as hydrothermal vents, etc., we would search for simplistic non-linear filters, etc. (such as EKF or UKF) but keeping the SWMD deduplication core.

• Distributed implementation: although our network relies on a fusion center for the implemented version, we would like to investigate a fully distributed version where neighbouring nodes work together and enhance the data using gossip protocols— which is more robust in case of single-failure points as experienced in sparse or mobile UWSN.

Author contributions

S.A.C, F.A and R.K conceived the experiment(s), S.N., S.A.C, S.A, F.A and M.A.K conducted the experiment(s), S.N., S.A.C, S.A, F.A and M.A.K. analysed the results. All authors reviewed the manuscript.

Funding

Princess Nourah bint Abdulrahman University Researchers Supporting Project number (PNURSP2026R239), Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia

Data availability

The OMNeT++ simulation code, synthetic datasets, and evaluation scripts are publicly available at https://www.kaggle.com/datasets/alexjhon82/under-water-sensor-dataset . The repository includes a detailed description and attributes of the underlying dataset.

Declarations

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1038/s41598-026-56139-8.