Early fusion of laser and acoustic features for human orientation detection in non-line-of-sight environments

Abstract

For search and rescue, security, and autonomous systems, determining the orientation of a human presence outside the line of sight is as crucial as detecting them. Knowing the human’s orientation plays a crucial role in determining the intervention strategy. In this study, data were obtained from scenarios involving human orientation outside the line of sight in a set of experimental settings. The original dataset was created using laser and acoustic chirp signals for the scenarios created to detect human orientation (behind, front, left, right). Data obtained from multiple sensors were correlated at the feature level using early fusion. A dataset with 42 features was created by combining two sensor sources, each with 21 features. Human orientation classification yielded strong performance under the adopted controlled experimental conditions using machine learning methods, artificial intelligence methods, and the proposed LAO-Net (Laser–Acoustic Orientation) model. Furthermore, explainable artificial intelligence analysis revealed the features that influence human orientation detection. The results of the study indicate that the fused laser-acoustic representation achieves strong human orientation detection performance under the experimental conditions evaluated. This study contributes to the limited literature on detecting human orientation in environments outside the line of sight.

Introduction

Detecting people or objects in a non-line-of-sight (NLOS) environment presents various challenges. Obtaining information about environments outside the sensor’s field of view is crucial today, especially in buildings and post-disaster debris. Line of sight often contains numerous obstacles, such as walls, columns, concrete blocks, and metal surfaces. These obstacles alter the propagation characteristics of the signals for sensors to detect, weaken the signal, and create scattering and reflection problems. Therefore, obtaining information about people or objects in NLOS environments becomes difficult.

Attempts to identify targets behind obstacles and to access details such as their location, posture and orientation are also gaining importance. Determining the orientation a person is facing in an NLOS environment is particularly important for providing accurate intervention to a victim trapped under debris. In indoor security scenarios, knowing human orientation is also crucial for autonomous driving systems.

This study focuses on estimating the orientation of a person behind an obstacle in an NLOS environment (behind, front, left, right) using data fusion of laser and acoustic signals. Features were generated from laser-based time-of-flight measurements and acoustic chirp signals. These features were combined using early fusion and estimation using Machine Learning (ML) and Artificial Intelligence (AI) models. It aims to contribute to the limited body of research in this field.

NLOS detection and application areas

NLOS detection attempts to obtain information about people or objects in an area that cannot be directly observed through indirect reflections. It focuses on obtaining information about the object’s presence and location. Various sensors are used to detect people or objects not in line of sight. These include different devices-sensors such as laser, acoustic, radar-microwave, Wi-Fi, and infrared¹. Because each sensor has different characteristics when obtaining information about the environment, it provides a variety of information. This information is also provided by data on propagation properties and noise. In recent years, ToF (Time-of-Flight), LiDAR (Light Detection and Ranging), and single-photon detector technologies have led to an increase in interest in interpreting information about scenes behind obstacles. They are used to interpret information obtained from measurements of scenes behind obstacles². An example NLOS environment is shown in Fig. 1.

Fig. 1.

An example of an NLOS environment.

NLOS detection plays a role in defense systems, search and rescue operations, indoor security applications, autonomous driving vehicles, and robotic applications^2–6. Detecting people or objects is valuable in scenarios where line of sight is limited or lost. This increases the need for reliable, fast, and portable detection systems for human or object information in NLOS environments.

Human detection in NLOS environments: current status and challenges

Obstacles in an NLOS environment block signal paths, making it difficult to obtain information about human presence. Various technologies and carrier sensors have been developed to overcome these challenges. One of these technologies is UWB (Ultra Wide Band) technology. It has a wide frequency range from MHz to GHz. In an NLOS environment, it generates short-duration pulse signals in these frequency ranges. It analyzes the reflection and refraction properties of the signals emitted into the environment. It aims to identify moving or stationary human presence in the environment. UWB receivers can detect small fluctuations such as breathing and movement. This demonstrates their effectiveness in extracting information about human presence⁷. In an NLOS environment, information about arrival times can be compared by measuring time differences. By achieving high time resolution, the path and reflected paths are distinguished. In recent years, deep learning methods have also been used to detect human presence from UWB signals⁸.

Radar and LiDAR systems are among the methods used for human detection in NLOS environments. Radar technology produces high-frequency electromagnetic signals. The signals bounce off obstacles in the environment. Values such as the arrival time and intensity of the echo signals are calculated. These values reveal information about the human being. The human body always reflects and absorbs radio waves. Breathing and heartbeat cause changes in radar signals. In this way, information about the human presence is obtained⁹. The arrival angle and temporal differences of the radar signals provide information about the approximate location of the target. LiDAR, on the other hand, sends laser beams and measures the ToF information of the light reflected from the beams. Changes in the reflected signal provide information about the human, while the flight time and beam reflection data provide approximate values about the location¹⁰.

One of the systems used for human detection is GNSS (Global Navigation Satellite System) and RTK-GNSS systems. These systems are satellite-based positioning systems. These systems are used to determine human location information rather than detecting humans. GNSS signals are sent via satellite. The signals reaching the receiver reach the receiver indirectly, which causes the signal to give an incorrect location. RTK-GNSS (Real-Time Kinematic) increases the accuracy of GNSS signals. It compares the phase measurements between the reference station and the mobile sensor. This allows the human location information to be given with very low error¹¹.

There are several challenges encountered in human detection in NLOS environments. These challenges, along with the limitations they create, negatively impact target detection performance. These limitations are grouped under specific headings and listed in Table 1.

Table 1.

Problems encountered in human detection in NLOS environments.

Limitations	Explanation-rationale
Limited real-world validation12–14	Many studies rely on simulations or controlled environments, which limit their validity in outdoor environments. In NLOS environments, which contain diverse environmental conditions, this weakens confidence in highly complex real-world environments
Sensor-specific focus15–17	Research has shown that they rely on single sensors (Radar, LiDAR, GNSS). This prevents and limits the ability to account for information revealing attributes from different sensors
Computational complexity14,18	The methods and artificial intelligence models used for NLOS detection are generally computationally expensive. This limits their adaptability to real-world conditions. This creates limitations in time-critical applications
Limited Dataset Variety6,19,20	The datasets used in detection studies using machine learning and artificial intelligence models are quite limited. These datasets have low generalizability to urban environments and are less robust in underground debris studies. One of the limitations is the lack of a dataset suitable for changing geographic and environmental conditions
Limited long-term stability21,22	Some sensors cause error accumulation in location determination methods in long-term, degraded environments. This degrades positioning accuracy and reduces reliability
Inadequate consideration of dynamic obstacles12,23	Studies focus on static and simplified scenarios. Studies have neglected the dynamic impact of mobility. This suggests a focus on standard scenarios and inadequate consideration of dynamic obstacles
Hardware and cost constraints16,24	Sensors used in NLOS environments typically utilize high-precision tools such as LiDAR and Radar. This suggests that hardware constraints and cost-limited applications are being adopted

The technologies used for human detection and their associated limitations are outlined. In addition to human detection, determining the human’s orientation in NLOS environments is also crucial. Determining the human’s orientation will provide a guide to determining the type of operation to be conducted against the target.

Problem definition and motivation: human orientation detection in NLOS environments

There are various studies on human presence. Human detection is a critical topic, particularly for search and rescue, autonomous driving, and localization. Furthermore, knowing a person’s posture after detection is also crucial for complex scenarios. Determining which orientation a person is facing after detection is a particularly important research topic for search and rescue teams. The original problem addressed in this study involves estimating human orientation (behind, in front, to the left, to the right) in an environment without a line of sight. The focus is on classifying the orientation a person is looking using laser and audio signals, independent of the person’s posture (standing, sitting, crouching).

Determining a person’s orientation in an environment without a line of sight is a challenging process. Reflective obstacles (walls, aluminum, etc.), the shape of the environment, multiple reflections, and ambient noise can cause distortions. Orientation information can be directly or indirectly manipulated by these obstacles. Various characteristics of laser and acoustic signals—such as amplitude, time delay, reflection distribution, and frequency content—may not provide sufficient environmental information. Classical thresholding and statistical methods fall short in such situations. It is necessary to extract features from these characteristics and employ machine learning methods.

Considering the studies in the literature, it appears that they focus on human presence. A few rare studies have focused on estimating the direction of moving people or objects²⁵. It appears that most current studies rely on single-sensor configurations, which may limit the amount of available supplementary information for orientation estimation.

In this study, a new dataset was created using early fusion methods using different features of laser and acoustic chirp signals. ML/AI methods were used to determine human orientation from this dataset. This study aims to contribute to this limited area of the literature.

In many scenarios where there is no line of sight, detecting human presence may not be sufficient for effective decision-making mechanisms. Obtaining human orientation information can provide additional benefits for various application scenarios. Based on this need, this study investigates whether combining laser and acoustic signals at the feature level to enrich information in the hidden scene can support human orientation classification in line-of-sight-restricted environments.

Data fusion and artificial intelligence perspective

Human detection in NLOS environments has generally been performed based on a single-sensor approach. The sensitivity of material properties in the environment often negatively affects the performance of a single sensor. Radar and IR-UWB-based systems can be examples of this. These sensors depend on the properties of the obstacle material and the geometric shape of the environment. Wall thickness, humidity, and the presence of metallic surfaces can cause signal reflections to increase or decrease. Furthermore, systems using only thermal-IR cameras are sensitive to factors such as ambient temperature, sunlight, and airflow. These types of sensors lose reliability in environments such as under debris, smoke, or dust. Single acoustic signals are negatively affected by background noise and echo reflections. Laser-only systems, on the other hand, can provide unstable results due to factors such as low SNR, photon statistical information, and narrow field of view. Despite the strengths of these sensors, they are not sufficient when used alone in complex environments^10,26.

While studies using data from single sensors are common, studies using sensors with different features together are gaining increasing importance. There are studies that perform decision-level data fusion using different sensors, such as radar-thermal, radar-IR, and gas-acoustic, together. Each of these sensors produces a decision. The generated decisions are combined with specific logical rules or voting schemes. This late fusion process is easy to design and interpret. However, it cannot provide rich information at the feature level. Therefore, it does not provide the flexibility to suppress inconsistent or weak signals. This may cause decision-level fusion of signals to be insufficient to achieve the desired separation when determining human orientation in an NLOS environment²⁷.

Evaluating signal features using machine learning methods using early fusion methods can be a powerful tool. Early-level fusion of features from different sensors prevents the loss of sensor-specific features. This allows for the creation of a common feature pool. The resulting early fusion data makes it possible to access target information using ML/AI methods. It retains complementary information such as distance and time delay patterns in laser signals, frequency information, reflections, and energy distribution in acoustic signals. This information represents detailed information for determining human orientation in an NLOS environment. Studies on signs of life in an NLOS environment have been conducted in the literature using CNN, LSTM, and various machine learning methods. Some of these studies were conducted on single sensors. In a limited number of fusion processes, decision-level fusion was also used. While studies at the early fusion level are limited, they are mostly focused on human presence or location information. In light of this information, the contributions of this study are outlined in the next section^28,29.

Contributions of this study and article organization

In this study, optical lasers and acoustic chirps were used to obtain information about objects in the NLOS part of the experimental environment. The aim was to determine human orientation (behind, front, left, right) regardless of the person’s posture (sitting, crouching, standing). Features from each source were extracted, and features from the two sources were combined using early fusion. Orientation was determined using ML/AI models. Model comparisons were conducted. The performance impact of the features was determined using explainable artificial intelligence.

The main contributions of this study can be summarized as follows:

• The study investigates the detection of human orientation (behind, front, left, right) in an NLOS environment, regardless of their posture (standing, crouching, sitting). The study aims to contribute to the limited literature on human orientation detection.

• Data fusion was applied to laser and acoustic chirp signals from two different sources. In the NLOS environment, studies examining human orientation through the fusion of data from multiple sources remain limited in the literature.

• An experimental setup was developed to simulate the key characteristics of a NLOS environment under controlled laboratory conditions.

• This study compared the results obtained using various machine learning and artificial intelligence methods against various criteria. This comparison may serve as a preliminary guide for model selection in future studies in this field.

• By taking 21 features from each data source (laser, acoustic), a dataset consisting of 42 features was created using early fusion.

• The results suggest that early fusion yielded a useful feature representation under the evaluated conditions; however, its specific advantage over single-modality inputs was not quantified in the present study. Additionally, since prior studies specifically focusing on human orientation detection in NLOS environments remain limited, this study aims to contribute to this field and provide a basis for future research.

• Feature analysis was conducted using explainable artificial intelligence to examine the relative contribution of the extracted features to model predictions.

The first section of the study includes an introduction and descriptive information about the NLOS environments. The second section addresses previous work in this area. The third section covers the creation of the experimental setup, data acquisition, preprocessing stages, the ML/AI models used, and a detailed material-method section of the proposed model. The fourth section presents experimental results and comparative data. The fifth section discusses the obtained results and presents the effects of features on performance using explainable artificial intelligence. The sixth section evaluates the results of the study.

Related works

NLOS detection techniques

Various technologies are used for human/object detection in NLOS environments. While each technology has a different system approach, the primary goal is to obtain information about entities outside the field of view. The studies conducted for this purpose and their related work are detailed in Table 2.

Table 2.

Non-line-of-sight detection technologies and information about the system used.

Detection type	Environmental robustness	Sensor fusion effectiveness	Vital sign sensitivity	Operational feasibility
High accuracy through CNN + LSTM fusion with radar and infrared data30	Robust in post-earthquake rescue scenarios with congestion	Effective decision-level fusion improves detection reliability	Focuses on signs of life under debris	Moderate complexity; suitable for rescue environments
UAV-borne radar detects vital signs through obstacles31	Operates in low-visibility and congested environments	Radar fusion with other UAV sensors	Remotely detects breathing and heartbeat	Mobile, adaptable UAV deployment
UAV system combines radar and lidar for SAR support32	Effective in complex urban environments	Radar and lidar fusion with Artificial Intelligence (AI)	Detects breathing and pulse movement	UAV-based; real-time mapping
Low-cost thermopile sensor detects survivors under debris33	High detection accuracy with low false alarms	Single sensor; no fusion	Detects presence via thermal signals	Lightweight, low-power, robotic deployable
Testing millimeter-wave radars for vital signs extraction 34	Range resolution across frequencies	Radar only; no fusion	Detects breathing and heartbeat	Laboratory and field testing
Critical review of emerging human sensing Technologies35	Evaluates radar, AI, drones, gas, acoustics, and optics	Discuss fusion and its limitations	Detects human presence in disaster areas	Focus on SAR mission impact and future trends

The table focuses on the detection of people under debris and outside the field of view of various technologies. Studies on human detection utilize systems such as lidar, radar, laser, optical, and imaging. Some studies rely on gas and acoustic assessments in the disaster area. Table 2 presents the advantages of different detection technologies and their application methods. Systems such as radar and lidar offer high spatial resolution. Acoustic systems, on the other hand, offer lower cost and have shown to be more sensitive to complex environments. Using a single sensor increases sensitivity to noise and material effects. Deep learning and fusion methods have begun to be used to overcome these limitations.

Studies on human detection

Radar, infrared, and acoustic sensors have physical properties and operational limitations. This has not yet addressed the specific problem of accurate target detection and localization under NLOS conditions³⁶. However, existing studies have been conducted using these technologies under various environmental conditions. These studies compare the technologies used, their accuracies, and limitations, and provide information about the gaps created^37,38. Studies offer different perspectives on accurate detection under NLOS conditions. While some studies advocate optimal sensor combinations and fusion strategies, others advocate passive infrared and acoustic sensing^27,39. The lack of a standard performance metric and dataset in most of these studies further complicates the process⁴⁰. Studies show that infrared provides robust information based on light variations. Acoustic sensors, on the other hand, are reported to capture useful audio data for detection and localization in occluded environments^27,41.

Human detection in NLOS environments using laser signals has been studied in post-disaster search and rescue operations, military applications, and autonomous systems. There are two types of NLOS-based laser systems: optical methods (ToF, LIDAR, interferometry) and electromagnetic approaches (radar/RF). Optical methods provide high-resolution spatial information. Radar/RF focuses on wall penetration and vital signs detection. Single-Photon Avalanche Diode (SPAD) and commercial LIDAR have a millisecond response time. Information is obtained from the histogram of the scattered light time and photon count⁴². This process progresses as the laser spot, the interface, and the hidden target, and photons are counted as the hidden target, the interface, and the detector⁴³. The histogram of photons detected by the detector can provide important information about objects in NLOS environments. Some studies on human detection are listed in Table 3.

Table 3.

Studies conducted on human detection and evaluation of the studies.

Detection accuracy	Operating range	Spatial resolution	Power consumption	Environmental robustness
Medium accuracy; limited to complex backgrounds35	Short to medium range; affected by debris density	Limited spatial resolution due to clutter	Medium; Depends on sensor type	Sensitive to environmental noise and obstacles
High accuracy (89% overall); Deep learning improves detection44	Effective on 10 cm concrete walls	High resolution with pulse radar and CNN	Medium; Portable system design	Robust against material interference with AI support
Improved accuracy with AI integration45	Variable; sensor and scenario dependent	Medium; enhanced by multi-sensor data fusion	Medium; UAV and sensor power constraints	AI helps mitigate environmental challenges
Reliable detection using multi-sensor fusion46	Short to medium range; robot mobility limits	Medium; combined visual and thermal sensors	Low to medium; Low-cost sensors prioritized	Limited by terrain and vibration
High accuracy through multi-modal fusion and deep learning47	Medium range; suitable for urban environments	High spatial resolution from EO/IR and RF fusion	Optimized for low swap-C on mobile platforms	Robust in urban clutter and variable conditions
Effective detection of breathing and pulse via radar32	Medium range; UAV-borne sensors	Medium; fusion with lidar improves mapping	Medium; UAV power limits mission duration	Robust to limited visibility and urban clutter
Effective survival detection with multi-sensor drone48	Medium range; Infrared Ultra-Wideband (IR-UWB) radar and laser rangefinders	High resolution with lidar and camera fusion	Medium; UAV energy constraints	Robust in complex post-disaster environments
High accuracy (87.9% correlation with vital signs)49	Short to medium range under rubble	High resolution for multiple targets	Medium; Radar power optimized	Sensitive to target pose and debris variability
Reliable detection using PIR and heart rate monitoring50	Short range; robot proximity required	Low to medium; sensor fusion improves	Low; Designed for field deployment	Limited by environmental noise and debris
Improved detection with dual-frequency IR-UWB radar51	Medium range; improved penetration	Medium; bandwidth increases resolution	Medium; Dual-frequency operation	Robust to weak vital signals under rubble
Improved detection accuracy with wavelet processing52	Medium range; effective clutter suppression	Medium; improved signal extraction	Medium; Computationally efficient	Robust in complex demolition scenes
Effective detection combining CO2, thermal, and audio53	Short to medium range; sensor synergy	Medium; sensor fusion improves localization	Medium; Sensor types vary	Robust to debris with a multi-sensor approach
Reliable target detection through various walls54	Medium range; handheld UWB radar	Medium; real-time detection	Medium; Handheld design	Robust to different wall materials
Accurate heartbeat detection with IR-UWB transceiver55	Short to medium range; multi-survival	Medium; pulsed Doppler phase detection	Very low power; Optimized for low power	Robust to noise with Gaussian pulse shaping
Effective life detection with microwave system56	Medium range; L-band frequency	Medium; frequency shift detection	Medium; Mechanical and antenna design	Robust to metabolic and cardiac fluctuations
Reliable life signs with wireless sensor network Detection57	Medium range; multi-sensor integration	Medium; sensor network improves detection	Medium power; Wireless communication	Robust in field and harsh environments
Effective life-detection radar for multiple applications58	Medium range; electromagnetic wave penetration	Medium; physiological characteristic detection	Medium power; Radar optimized	Robust in a variety of disaster and military environments

Table 3 presents studies on human detection using different sensor technologies. These studies focus particularly on radar and IR-UWB-based approaches to detecting life behind walls and under debris. As noted in these studies, human presence detection is achieved with accuracy exceeding 90%. Most studies utilize a single sensor, while studies using multiple sensors are quite limited.

Various technologies have been used to develop detection systems for human presence and location NLOS environments. Examples include laser, radar, ultrasonic systems, sonar, acoustic signals, drone-based systems, autonomous vehicles, and search and rescue robots. These technologies are broadly classified into three groups: optical-based systems (Light Detection and Ranging—LIDAR, ToF), electromagnetic-based systems (Radar, Radio Frequency-RF), and acoustic-based systems. These systems attempt to access target information using image processing, signal processing, machine learning, and artificial intelligence methods^30,35,59,60. This section reviews literature on object and human detection in NLOS environments. The systems, detection methods, fusion approaches, and the use of artificial intelligence models used in these literature studies are presented. Furthermore, trends in research in this area and emerging limitations are outlined.

Multi-sensor fusion approaches and human detection with artificial intelligence in NLOS environments

Data fusion involves combining information from different data sources into a single data source. Various methods are used for this approach. In this study, data fusion approaches were used to capture data from multiple sensors in NLOS environments. In addition to human detection using data fusion, localization studies using multiple sensors are also being conducted. A literature summary of these studies is provided in Table 4.

Table 4.

Development studies with data fusion in NLOS environments.

Performance	Robustness in NLOS conditions	Computational load and suitability for real-world environments	Sensor fusion and modality complementarity	NLOS ıdentification and mitigation approach
Achieved 1.46 m error using deep residual fusion61	Adaptive zoning of NLOS regions improves robustness	Medium complexity with deep learning and clustering	Frequency, power, and time-delay fingerprinting	Iterative clustering and classification for NLOS region separation
53.66% improvement when integrating GNSS with IMU/camera; 22.14% with UWB62	Factor graph optimization improves stability in GNSS obstructions	High due to factor graph optimization	IMU, camera, GNSS, and UWB	Sensor factor residual models and linkage for NLOS mitigation
Location accuracy increased by 71.98% compared to UWB alone in indoor environments63	Robust in challenging indoor NLOS environments using factor graphs	Medium with factor graph fusion and DWKNN	UWB, IMU, and geomagnetic sensors	Dynamic weighting and factor graph fusion to address UWB faults
Average localization error 1.2 m in urban and 1.5 m in forested environments64	Robust to sensor noise and environmental challenges in GPS-obscured areas	Medium with loop closure and multi-sensor fusion	IMU, LiDAR, and visual sensors	Loop closure and sensor fusion mitigate NLOS effects
Sub-decimeter accuracy (0.086 m RMSE) with NLOS correction and FGO65	Factor graph optimization improves robustness in sensor geometry	High due to factor graph and gray prediction model	UWB and LiDAR SLAM	Grey prediction model for NLOS correction and real-time exclusion
Location accuracy increases with the number of reflection paths66	Spatial multipath fusion provides good robustness in complex environments	Medium with signal noise subspace decomposition	Multi-array antennas and spatial multipath data	Spatial multipath fusion for NLOS mitigation
Superior accuracy during obstructed LOS using active/passive fusion67	Improved robustness by fusing active and passive radio measurements	Medium with particle-based sum product algorithm	Active and passive radio measurements	Probabilistic data association for NLOS and OLOS scenarios
Location accuracy increased by 69.16% compared to UWB alone68	Optimized particle filter with sparrow search algorithm improves robustness	High efficiency with reduced particle count	UWB and LiDAR	ISSA optimization reduces particle corruption and NLOS errors
Robust localization in fog using LiDAR and RADAR fusion69	Fusion reduces sensor degradation in low visibility	Medium with clustering and correspondence matching	LiDAR and RADAR	DBSCAN line segmentation and sensor
Improved detection of partially obscured objects through fusion70	Radar and LiDAR mitigate each other’s limitations	Medium with sensor fusion	LiDAR and Radar	Utilizing complementary frequency spectrum
NFOV target estimation combining optical and acoustic sensors71	Recursive Bayesian estimation with probability construction	Medium with Bayesian fusion	Optical and acoustic sensors	Probabilistic fusion for NLOS targets
Landmark-assisted fusion Positioning accuracy improved by 24.4%72	Kalman filter with cumulative error analysis for Lidar noise	Medium with Kalman filtering and landmark assistance	UWB and 2D LiDAR	Signal-aided fusion with noise covariance adaptation
2 × accuracy improvement with adaptive noise covariance73	Variance and power envelope-based NLOS identification	Low complexity with adaptive covariance adjustment	UWB and IMU	Adaptive noise covariance and NLOS identification

Studies have shown that multi-sensor and multi-attribute fusion in NLOS environments increases accuracy and stability. Sensors such as LiDAR, RADAR, optical, and acoustic sensors appear to have a complementary effect. Furthermore, machine learning, artificial intelligence, and deep learning techniques are emphasized for improving accuracy. However, due to the lack of datasets and limitations of real-time applications, these studies are not widely used.

Furthermore, a study has demonstrated that by detecting falls, recognizing daily activities, and identifying posture, it can provide useful insights for existing research⁷⁴. Another study noted that spatiotemporal feature fusion could contribute to distinguishing a person’s state in fall detection⁷⁵. In a separate study, sensor-based systems utilizing wearable technologies were employed. Here, an approach was proposed for applications capable of detecting daily body activities and issuing fall alerts⁷⁶. Studies on human posture recognition have indicated that feature representations can be used to distinguish between different human bodies⁷⁷. These studies are generally conducted in line-of-sight, vision-based, or wearable-sensor settings rather than in NLOS laser–acoustic environments. These studies support the broader premise that human orientation- and posture-related information can be learned from feature representations. Consequently, this further motivates our efforts to investigate laser-acoustic data fusion features for human orientation in NLOS environments.

Research gap and positioning of this study

A review of the literature reveals that studies using laser and acoustic sensors together remain limited. Applications attempting to estimate human orientation by fusing acoustic and laser signals are limited in the literature. This indicates that human orientation detection in NLOS environments has been relatively under-researched in the literature. Our study aims to address this gap by investigating the fusion of laser and acoustic features for human orientation detection. This study examines the use of laser and acoustic signals fused at the feature level to detect human orientation, with the aim of contributing to this area. In this context, the contributions of this study to this literature gap can be listed as follows:

• Multi-sensor fusion studies are increasingly widespread and appear in the literature. This study utilizes signals from multiple sensors through fusion.

• There are a limited number of studies in the literature using the early fusion method. In this study, the features of laser signals and acoustic chirp signals are combined, preserving the sensor characteristics through the early fusion method.

• While there are studies in the literature on human detection, research specifically focused on human orientation detection remains limited. This study aims to contribute to this field.

• The results obtained with the early fusion method using different ML/AI methods are compared. This research allows for comparisons between fusion and machine learning methods and artificial intelligence methods, providing guidance for future studies.

• A new artificial intelligence model used to detect human orientation has been presented and evaluated. This model provides a comparison with other ML/AI models.

• Using explainable AI techniques, the characteristics of laser and acoustic signals associated with orientation estimation are analyzed.

This study is conducted to address the following research questions:

• Can fusion of laser and acoustic features support four-class human orientation classification in non-line-of-sight environments?

• How effective are conventional machine learning/artificial intelligence models and the proposed LAO-Net under the adopted experimental protocol?

• According to Shapley-based explainability analysis, which fusion features contribute most strongly to human orientation prediction?

Materials and methods

Experimental setup

A two-room laboratory environment, with sound and light insulation, was designed for the system’s installation. The room measures approximately 2.6 × 3 m. The ceiling of the room is covered with furniture material. Acoustic insulation is provided using pyramidal foam. A barrier was constructed to create a 1.6 × 2 m partition inside the room. The interior surfaces of the hidden stage are covered with pyramidal foam. A pleated blackout curtain is used to suppress laser signal reflections and absorb ambient light. The resulting room setup is shown in Fig. 2.

Fig. 2.

Experimental room setup where data acquisition was carried out.

An acoustic audio signal system was installed to capture audio data. The system utilizes a mobile platform with a 1-m horizontal stroke. Mounted on the platform are eight microphones and an eight-speaker array. Microphones are evenly spaced on the left side of the system, and speakers are evenly spaced on the right. The microphones and speakers are spaced 8.8 cm apart. This allows for horizontal and vertical scanning.

The ECM8000 was used as a sound measurement microphone. This equipment ensures equal detection of sound from all directions and provides excellent performance in the 15–20 kHz range. The sound systems utilize an S-S10 speaker component, which provides high-frequency reproduction. The ADA8200 (8 × 8 preamp) was used as the interface for the microphones. This ensures high-quality recording in this multi-microphone system. The preamp-sound card connection is made via fiber optic. The UMC1820 was used as the USB sound card, which provides high-quality input and output. The MX 8200 (8-channel amplifier) was used to ensure the correct power output for each speaker in the system. The hardware devices and specifications used in the experimental setup are detailed in Table 5.

Table 5.

Hardware used for data acquisition in the experimental setup and their features.

Hardware	Function	Reasons for use
ECM8000	Records ambient sound with reference accuracy	For reliability of acoustic measurements
S-S10 TW Tweeter	Reproduces high frequencies	For complete testing of the system frequency response
ADA8200	Amplifies and digitalizes eight microphone signals	For multi-channel measurements
ADAT Connection	Transfers digital signals without loss	For noise-free and synchronized data transmission
UMC1820	Transfers signals to a computer and records	For multi-channel data recording and analysis
MX 8200 Amplifier	Drives speakers with the required power	For accurate sound pressure measurements

The hardware connections of the acoustic audio signal system used in the experimental setup are shown in Fig. 3.

Fig. 3.

The layout of the acoustic audio signal system on the experimental setup and its operating logic.

In the laser system, the laser beam was transferred to the galvo mirrors via a Polarizing Beam Splitter (PBS). This beam was directed to the reflective relay surface. The beam then reflected from the relay surface and reached the hidden scene. The returning photons were then accepted back into the system and detected with a time stamp by a Single-Photon Avalanche Diode (SPAD) camera. Photon counting was performed and collected using the Time-Correlated Single-Photon Counting (TCSPC) module. The temporal profile of the signal was generated by histogramming the arrival times of the returned photons. The Time-to-Digital Converter (TDC) combination achieved nanosecond/millisecond precision. The laser system structure used in the experimental setup is shown in Fig. 4.

Fig. 4.

Schematic view of system components.

Laser and acoustic audio signals were positioned together in the experimental area. The system image is shown in Fig. 5.

Fig. 5.

General system view created by combining the laser and acoustic systems.

Data collection

The data collection process was carried out in accordance with relevant guidelines and regulations. The experimental protocol involving human participants was reviewed and approved by the Firat University Social and Human Sciences Research Ethics Committee (approval number: 12.01.2024–21,330). All participants were informed and their consent was obtained before the data collection process.

The distance between the acoustic source-receiver array and the relay surface is 120 cm. The signal’s transmission and reception timing and scanning cycles are configured to ensure synchronicity.

In the experimental setup, the data acquisition process was carried out according to a specific workflow. These are;

1. Before the subject is placed in the room, a noise recording of the empty stage is taken.

2. Each of the eight speakers emits a chirp sequentially at 2-s intervals. All eight microphones are on and recording simultaneously. Each chirp is emitted linearly for 1 s (2–20 kHz).

3. After each speaker’s sound is recorded individually, all speakers emit a chirp simultaneously. All microphones record.

4. Steps 1–3 are taken on the stage without the subject present. They are recorded as white noise.

5. After the noise recording is taken, the subject is placed on the stage. Data are collected for steps 2–3 based on 12 posture combinations: seated, crouched, and standing on the hidden stage, facing behind, facing front, facing left, and facing right. Laser scanning is performed for each position in the same manner, and the time histogram of the returned photons is counted using the SPAD + TCSPC chain.

In the study, samples were taken from eight subjects in the seated, standing, and squatting positions. Laser and audio data were collected for each orientation class, labeled as behind, front, left, and right. Each laser dataset contains 16,220 rows. Each row represents the number of photons received in 1 ms. When divided into 36 windows, 450 orientation samples were extracted from each laser dataset. A single acoustic audio signal dataset corresponds to the laser data. The total number of fused samples for each orientation class is 10,800. A consistent random-sampling-based splitting was performed for the proposed LAO-Net model and the other evaluated models. Because multiple 36-sample windows were derived from the same original laser scan, these window-level samples may not be fully independent from one another. Twenty percent of the fused dataset was set aside as an independent test set. The remaining 80% was used as the model development set. In model development, fivefold cross-validation was applied, where one fold was used for validation and the remaining folds were used for training in each fold. The generated dataset is detailed in Table 6.

Table 6.

Number of laser and Acoustic samples obtained from the subjects orientation.

Behind		Front		Left		Right
Laser	Acoustics	Laser	Acoustics	Laser	Acoustics	Laser	Acoustics
24 (450)	24	24 (450)	24	24 (450)	24	24 (450)	24
10,800		10,800		10,800		10,800

After the laser data was sent, the return photon count was recorded. It was then waited, sent again, and the process repeated. The laser device in the prepared experimental setup was moved along the rail track, sending and receiving laser data. This process continued for 16 s. Sample data from this sample is shown in Table 7.

Table 7.

Sample laser data taken within 12 ms.

Duration ms	Number of photons
1	2937
2	2431
3	59
4	71
5	67
6	75
7	7662
8	2355
9	54
10	93
11	84
12	74

Table 7 shows the number of photons returned from the transmitted laser data. The number of photons received during the first 2 ms of the transmitted laser data and the incoming data is shown. After this, the number of photons obtained from the medium continues with a 4 ms delay. It is seen that a sample of data is sent and received every 6 ms.

Data preprocessing

The audio and laser files were combined and saved under the same name. Laser data is in csv file format, while acoustic chirp signals are in wav file format. Laser data contains 16,220 rows of photon counts. Each row in the laser data indicates the number of photons arriving at 1 ms intervals. The audio files consist of an 8 × 2048 matrix. The raw SPAD-TCSPC laser data were normalized by subtracting the white noise collected before the experiment began. White noise is data collected from a room outside the field of view, indicating that the room was empty. Similarly, the audio data was subtracted from the white noise to form the audio data used in the study.

Laser data preprocessing process:

To capture local temporal variations in the photon count sequence, the entire laser recording was divided into 36 windows rather than being represented as a single data set. Each laser matrix has dimensions of 1 × 16,220. In the study, windows of 24, 36, 48, and 64 samples were generated, but because 36 provided the best results, it was decided to use 36 windows. The portion that was not fully divided was discarded. Each window was labeled to indicate the subject’s stance. However, multiple windows were obtained from the same scan, and these window-level samples may therefore retain some dependence. Accordingly, the reported findings should be interpreted within the adopted window-based evaluation framework rather than as results derived from a set of fully independent observations. The temporal graph of the laser data and the process of dividing this graph into windows are shown in Fig. 6.

Fig. 6.

Original laser signal on the left, Separation of the laser signal into 36 windows on the right.

Acoustic data preprocessing process:

The audio data, consisting of an 8 × 2048 matrix, contains acoustic chirp signals from eight microphones. The average of the eight microphones in this audio signal was converted into a 1 × 2048 matrix. Each audio file was labeled according to its orientation. The acoustic audio signal and the averaging process are shown in Fig. 7.

Fig. 7.

The 8-channel acoustic chirp signal is averaged and converted into a single-channel sound signal.

Feature extraction

To capture local changes in orientation information in the time series of the laser signal, it was divided into 36 × 1 vector windows. Each window in the signal was treated as a window-level sample for feature extraction, and 21 laser features were extracted from each segment. The acoustic chirp signal, on the other hand, was reduced to a single channel by averaging eight channels of data and has a 2048 × 1 vector. Twenty-one features were extracted for this acoustic segment. This window-based representation was used to preserve the local temporal structure of the laser signal. However, windows derived from the same scan may not be entirely independent.

The following features are calculated for each window (laser: 36 × 1 window; audio: 2048 × 1 average channel). All features are z-score normalized (leakage prevention) immediately before starting the training process. The features used and their information are listed in Table 8. The table includes the feature name, the effect this feature attempts to capture on the laser signal, the feature’s effect on the acoustic segment, and the feature’s equation.

Table 8.

Attributes used in the study and the effects of these attributes.

Group	No	Attributes	Laser effect	Acoustic effect	Equation
Basic statistical properties	1	feat_mean: Mean	Permanent shifts in reflection power	Average power of echoes
	2	feat_std: Standard deviation	Can increase with multiple reflections/complexity	Room modes/fluctuation increases with multiple echoes
	3	feat_var: Variance	Carries the same information as Std on a gridded scale	Conveys same information as std on a squared scale
	4	feat_median: Median: The middle of the ordinal data	Resistant to noise and outliers	Robust center tendency in noisy recordings
	5	feat_mad: Median Absolute Deviation	Resistant to single photon jumps/interference and specular peaks	Resistant to outliers such as sudden impacts
Amplitude/range & energy content	6	feat_min: Minimum values	Deep pits; ghosting/blocking traces	Absorption/cancellation zones
	7	feat_max: Maximum values	Peak scattering; bright spots	Strong early echo/resonance peak
	8	feat_ptp: Peak-to-peak:	Dynamic range; varies with orientation/material	Dynamic range; sensitive to source-target geometry
	9	feat_rms: Root-mean-square	Effective photon count level	Effective acoustic power
	10	feat_energy: Energy	Total power; energy in the scan patch	Total power; secondary echo intensity
Shape/trend	11	feat_slope: Linear slope:	Local increase/decrease trend in the scan	Power Trend within the window (early to late echo)
	12	feat_skew: Skewness	Asymmetric scattering (single dominant reflection)	Asymmetric echo distribution
	13	feat_kurt: Kurtosis	Spiked peak/multi-peak structures	Narrow resonances/sharp peak behavior
Time dependence	14	feat_ac1 Autocorrelation coefficients	Short-scale periodicity; scan pattern	Short delay echo dependence
	15	feat_ac2 Autocorrelation coefficients	Longer short-period effects	Room modes/early-mid echoes
Spectral summaries (Fast Fourier Transform-FFT based)	16	feat_spec_centroid: Spectral center of mass	Energy “center of gravity”; shifts with geometry	Energy shift to HF; material/distance effect
	17	feat_spec_spread: Spectral spread	Bandwidth; scene complexity	Bandwidth; echo richness
	18	feat_spec_entropy: Spectral entropy	Spectral non-uniformity; multiple scattering	Spectral non-uniformity; multiple echoes/scatter
	19	feat_dom_freq_bin: Dominant frequency component	Dominant “scan-frequency” component	Dominant frequency (Hz); position/resonance effect
	20	feat_dom_power: Dominant power	Strength of dominant component	Strength of dominant component
	21	feat_rolloff85_bin: 85% spectral roll-off component	Energy threshold; HF content limit	Energy threshold; limit of reach to HF

The features are grouped and detailed in the table. Different features were used to capture features that could impact performance in the study. Basic statistics, amplitude/range & energy, shape/trend, time dependence, and spectral features are grouped and presented in the table.

Data fusion

Fusion is a method used to obtain more reliable results by combining data from different devices. The data fusion method employed in this study aimed to obtain more reliable information by combining information from both laser and acoustic chirp signals. Directly combining raw signals and scaling them synchronously for NLOS is quite challenging. Therefore, the signals from each data source were combined after extracting their features. This feature vector was then merged into a single vector. After the laser data was divided into 36-sample windows, 21 features were extracted, while the acoustic chirp signal was extracted from the 2048 × 1 vector. These two 21-dimensional feature vectors were combined into a single 42 × 1 vector. A classification label was then added to this vector, creating a 43 × 1 vector. The fusion process created by combining the features of each signal data is called early fusion or feature fusion. The feature vector fusion is given in Eq. 1. Additionally, the early fusion process is given in Fig. 8.

Fig. 8.

Process of early-feature fusion of laser and acoustic chirp signal.

(laser feature vector) (acoustic feature vector) is combined and the feature vector is obtained with the following equation.

1

The combined dataset contains 43,200 examples. Twenty percent of these (8640 examples) were set aside as an independent test set, while the remaining 80% (34,560 examples) were used for model development via fivefold cross-validation.

Models used in the study

In this study, data obtained through early fusion was trained and tested with various ML/AI models. Class information was added to the laser and acoustic chirp signal features obtained through data fusion to create a data set with 43 × 1 vectors. The resulting data set was normalized with a z-score before model training. The models used in the study are presented in groups.

Decision Trees: Classification is a widely used method in prediction processes. Data are divided into groups by branching according to their attributes. Data grouped by branches and leaves are classified. There are three types of trees based on their branches and leaves: fine tree, medium tree, and coarse tree. Fine Tree: It consists of many branches and nodes. It tries to find complex patterns in the data⁷⁸. Since this model can perceive noise in the data as a pattern, it can cause overfitting in small data sets⁷⁹. Medium Tree: It has a reasonable number of splits and complexities. It is generally preferred in applications where interpretation is important and some complexity is normal⁸⁰. Therefore, it cannot capture all the nuances. Therefore, it performs less accurately than fine trees⁸¹. Coarse Tree: It has the least splits and branches among decision trees. It is more suitable for small data sets. It oversimplifies the data. Therefore, it may miss patterns. Its accuracy rate is lower than other decision trees⁸².

Naive Bayes: In cases where conditional independence exists, it performs inference based on class probabilities. The training and testing process is quite fast. Performance degrades when the condition is violated. It can be used in applications where resource requirements are low. However, its performance is relatively lower than other models. There are two types: Gaussian Naive Bayes and Kernel Naive Bayes. Gaussian Naive Bayes assumes that the distribution of features is normal. It is effective for continuous data⁸³. Kernel Naive Bayes, on the other hand, uses kernel functions. It handles nonlinear data distributions better than the Gaussian Naive Bayes method⁸⁴.

Support vector machines (SVM) It is a powerful supervised learning method for regression and classification. It is capable of processing high-dimensional data. It allows for the separation of nonlinearly separable data. It includes optimization methods such as gradient descent. Linear SVM performs well on linear separation of data⁸⁵. It cannot capture nonlinear relationships. Quadratic SVM, on the other hand, captures and classifies moderately nonlinear data. The polynomial kernel degree is 2. The probability of overfitting is higher. Cubic SVM has a polynomial kernel degree of 3. It has medium to high complexity. It is more successful in capturing nonlinear patterns. Its computational cost is higher than other SVMs. It is prone to overfitting. Fine Gaussian SVM: The kernel scale parameter is kept low. It has high flexibility. It can capture boundary details in the data well. It can distinguish boundaries with complex and small details. It can overfit when data is scarce. It has high computational cost. Medium Gaussian SVM: The kernel scale parameter is medium. It has medium flexibility. It is shown to be more stable and has a moderate computational cost. It is insufficient for structures containing very fine details. The kernel scale for Coarse Gaussian SVM is kept large. The degree of flexibility is low to moderate. It operates more stably in applications that are not noisy and do not contain excessive detail. The risk of overfitting is low^86,87.

K-nearest neighbors-KNN It is one of the most fundamental algorithms used in classification and regression problems. It is a nonparametric method based on edge neighborhoods. It produces quite good results on a dataset with good features. Fine KNN has local decision limitations. The K value is set close to 1. Its ability to capture detail is low and it is sensitive to noise. Medium KNN uses a moderate K value. It is more balanced between detail and balancing. It can overfit at high K values. Coarse KNN uses a high K value. It has better global trends. The noise effect is less. The overall classification trend is more stable. It misses small details. However, its computational cost is lower than others⁸⁸ Cosine KNN uses distance according to the similarity metric. It looks at the angle difference between two vectors and approaches based on their similarity. It works well on high-dimensional data and is not much affected by feature scale differences. Cubic KNN, on the other hand, establishes a connection using a specific number of neighbors and cubic distance. Calculation and interpretation are difficult. In Weighted KNN, weights are assigned based on the distance between neighbors. As the influence of nearby neighbors increases, the precision of the decision boundary increases. This reduces the detrimental effect of distant neighbors. The calculation is more complex. Distance normalization is important in this model⁸⁹.

Kernel-based extensions Linear models are transformed into nonlinear models by expanding the feature space. They exhibit strong performance when properly scaled. The SVM kernel transforms nonlinear data into a higher-dimensional space, making it separable. The kernel trick used in this method performs classification without transforming it into a higher-dimensional space. It transforms the data to linearize the decision class. In the logistic regression kernel method, the data is transformed using the kernel function. Nonlinear boundaries are created^90,91.

Ensemble methods These are machine learning techniques that combine multiple methods. The aim is to create a more robust, stable, and generalizable prediction model. The primary goal in such methods is to extract strong features by combining weak features. Overfitting and bias-variance tradeoffs are used to balance the challenges of the tradeoff. Boosted Trees sequentially train weak trees, correcting the errors of the previous tree. AdaBoost and Gradient Boosting are the most common. They achieve high accuracy on data with a deep tree structure⁹². Bagged Trees are also known as Bootstrap Aggregated Trees. They allow trees to be trained with random samples within the tree structure, generally reducing variance in unstable models⁹³. Subspace Discriminant applies the discriminant analysis model to feature subspaces, thus generating multiple models. Each model is trained with a different feature. Its performance is good on high-dimensional datasets⁹⁴. Subspace KNN trains a KNN model on randomly selected subsets. Each KNN model is trained with its own subspace, and the results are evaluated collectively. It is resistant to noise. It performs better on large datasets⁹⁵. RUSBoosted Trees are mostly used on imbalanced datasets. It aims to improve learning in classes with small sample sizes. It randomly deletes samples from classes with large sample sizes and applies boosting⁹⁶.

Artificial neural networks They exhibit quite good performance in solving nonlinear problems. They have a layered structure. They consist of an input layer, a hidden layer, and an output layer. Narrow Neural Network has a small number of neurons in a single hidden layer. It requires fewer computational resources. It is more suitable for applications with limited data and insufficient resources⁹⁷. Medium Neural Networks are medium-sized neural networks. They are ideal for applications that require a balanced complexity and computational power⁹⁸. Wide Neural Networks have a large-scale neural network structure. They contain a large number of neurons. They have effective learning and generalization properties. They perform better than other networks⁹⁹. Bilayered Neural Networks have a structure consisting of two hidden layers. This type of networks has two hidden layers. The two hidden layers provide higher performance by enabling better feature learning. As the number of hidden layers increases, the computational cost also increases¹⁰⁰. Trilayered Neural Networks have a three-layer structure. They have three hidden layers. They are quite successful in solving nonlinear problems. Their computational cost and complexity are high¹⁰¹.

While applying the ML/AI models used in the study, the flow diagram shown in Fig. 9 was applied.

Fig. 9.

Flow process for each model.

Proposed model LAO-Net (laser–acoustic orientation network)

A feedforward artificial neural-based multi-class classification model was designed to detect human orientation in a dataset consisting of 21 laser and 21 audio features. This model consists of three main steps: input data, a hidden layer, and an output layer. The input layer contains 42 features consisting of laser and acoustic chirp signals, while the hidden layer contains an artificial neural network model consisting of 30 neurons, and the output layer contains output information regarding human orientation. The primary reason for choosing a feedforward network in this study is the greater importance of simultaneously processing human information outside the field of view. Furthermore, under the evaluated experimental setup, this model achieved higher performance than the other tested models. The LAO-Net diagram is shown in Fig. 10.

Fig. 10.

Layered structure of LAO-Net.

Figure 10 shows the input layer of LAO-Net containing 42 features, while the output layer is intended to predict human orientation. The LM (Levenberg–Marquardt) method was used to optimize this neural network. The LM algorithm is better suited for shallow feedforward networks with a relatively small number of trainable parameters. It has been preferred due to its fast convergence. The use of a single hidden layer with 30 neurons enables the model to learn nonlinear relationships in the combined feature space. Therefore, it aims to maintain a compact architecture. Additionally, LM’s computational and memory requirements may be less suitable for larger datasets and deeper architectures. Twenty percent of the data was set aside as an independent test set, while the remaining 80% was used for model development with fivefold cross-validation. Training was conducted over 1,000 epochs. The hyperparameters used for model training are shown in Table 9.

Table 9.

Hyperparameters used for LAO-Net training.

Dataset	43,200
Partitioning strategy	Random sample-based partitioning
Test set	20% (8,640 samples)
Model development set	80% (34,560 samples)
Max epochs	1000
Optimization	Levenberg–Marquardt
Input data	21 Laser features + 21 acoustic chirp signal attributes
Output data	Human orientation (behind, front, left, right)
Hidden layer	30 neurons
Activation	ReLU
Hyperparameter selection	Predefined

Evaluation metrics

Twenty percent of the dataset was set aside as an independent test set. The remaining 80% was used for model development with fivefold cross-validation. Each fold served once as validation data while the remaining folds were used for training. The validation results are presented as the average across the five folds, whereas the final test performance was derived from the independent 20% test set. Global normalization was not applied to the entire dataset. Z-score normalization was applied separately within each cross-validation fold. The normalization parameters were calculated using only the training set of each fold. They were then applied to the corresponding validation and test sets. This was done to prevent data leakage. Several evaluation metrics were used to assess the performance of the training and testing processes of the models used in the study. These metrics demonstrate the performance of the study and are frequently used in the literature. Furthermore, the training and testing processes were analyzed separately using these metrics. Accuracy, Macro Precision/Recall, and F1 Score metrics were used. Furthermore, a confusion matrix was used to reveal correct and incorrect predictions in the test results produced by the models. The metrics used and their equations are presented in Table 10.

Table 10.

Evaluation metrics and their descriptions used in the study.

Metric	Equation	Feature
Precision		TPc: Number of true-positives
Recall		FPc: Number of false-positives
Accuracy		FNc: Number of false-negatives
Macro precision		N is the total number of samples
Macro recall		Overall performance; can be sensitive to the dominant class in case of class imbalance
Macro F1 score		Sensitive to false-positives; equal weighting between classes
Mean square error		Sensitive to missed (FN) samples; gives fair results in case of class imbalance

The evaluation metrics and equations listed in Table 10 were used to evaluate the results obtained from the models used in the study. These metrics are frequently encountered in the literature to allow comparison of the results obtained from the models.

To interpret the contribution of combined laser-acoustic features to classification outcomes, the global interpretation tools available in the MATLAB Classification Learner were used. A Shapley-based explainability analysis was conducted. Global Shapley importance was used to assess the predictive importance of features for the trained classifier. The resulting global importance ranking is based on the average absolute Shapley values of the predictors. Each Shapley value indicates a predictor’s contribution to the deviation of the predicted class score from the average prediction. The global ranking provides a summary of these contributions across the analyzed observations.

Experimental results

LAO-Net results

Fusion of laser and acoustic chirp signal characteristics was used as input to the LAO-Net for human orientation detection under NLOS conditions. The predicted output corresponded to one of four orientation classes. The quantitative results are presented in Table 11.

Table 11.

Results obtained with LAO-Net.

Accuracy	Macro-precision	Macro-recall	Macro-F1	Mean squared error (MSE)
99.98	99.98	99.98	99.98	0.0002

The results obtained from the fused dataset under the adopted experimental protocol are close to 100%. These findings demonstrate that the model exhibits strong performance compared to other models evaluated within the scope of the experimental protocol and the collected dataset. Figure 11 shows the graphs of the model’s training, testing, and validation processes.

Fig. 11.

Graphs of the results obtained from model training.

Figure 12 presents information regarding the training performance of LAO-Net. It shows the loss curve and the corresponding validation behavior over the epochs. The best validation performance was achieved at the 176 epoch. The relevant classification results are summarized using a confusion matrix.

Fig. 12.

LAO-Net training/validation and loss graph and graph of confusion matrix.

The models were trained using fivefold cross-validation on the dataset allocated for training and testing. Twenty percent of the dataset was set aside as an independent test set. Using this framework helps provide more reliable performance estimates during training and testing and can reduce the risk of overfitting. However, these results should be interpreted as dataset-specific findings obtained under controlled laboratory conditions rather than as evidence of broad real-world generalizability.

Comparative results

The results of the ML/AI models used in the study are given in Tables 12 and 13. In both tables, Accuracy% (Validation), Accuracy% (Test), Macro Precision% (Validation), Macro Recall% (Validation), Macro F1 Score% (Validation), Macro Precision% (Test), Macro Recall% (Test), Macro F1 Score% (Test) scores are given.

Table 12.

Good results obtained with ML/AI models.

Model type	Preset	Accuracy % (Validation)	Accuracy % (Test)	Macro precision % (Validation)	Macro recall % (Validation)	Macro F1 score % (Validation)	Macro precision % (Test)	Macro recall % (Test)	Macro F1 score % (Test)
Tree	Fine tree	99.92	99.81	99.92	99.92	99.92	99.81	99.81	99.81
SVM	Quadratic SVM	99.79	99.73	99.79	99.79	99.79	99.73	99.73	99.73
	Cubic SVM	99.89	99.81	99.89	99.89	99.89	99.82	99.81	99.81
	Fine Gaussian SVM	99.81	99.65	99.81	99.81	99.81	99.65	99.65	99.65
	Medium Gaussian SVM	97.87	98.65	97.89	97.87	97.86	98.65	98.65	98.64
KNN	Fine KNN	99.82	99.73	99.82	99.82	99.82	99.73	99.73	99.73
	Medium KNN	99.66	99.68	99.66	99.66	99.66	99.68	99.68	99.68
	Coarse KNN	97.77	98.63	97.77	97.77	97.77	98.63	98.63	98.63
	Cosine KNN	99.69	99.71	99.69	99.69	99.69	99.71	99.71	99.71
	Cubic KNN	99.48	99.48	99.48	99.48	99.48	99.48	99.48	99.48
	Weighted KNN	99.75	99.73	99.75	99.75	99.75	99.73	99.73	99.73
Kernel	SVM Kernel	99.77	99.66	99.77	99.77	99.77	99.67	99.66	99.66
	Logistic Regression Kernel	99.05	99.34	99.05	99.05	99.04	99.34	99.34	99.34
Ensemble	Boosted Trees	99.70	99.92	99.70	99.70	99.70	99.92	99.92	99.92
	Bagged Trees	99.98	99.91	99.98	99.98	99.98	99.91	99.91	99.91
Neural network	Narrow Neural Network	99.87	99.80	99.87	99.87	99.87	99.80	99.80	99.80

Table 13.

Results of models evaluated as low performing in the study.

Model type	Preset	Accuracy % (Validation)	Accuracy % (Test)	Macro precision % (Validation)	Macro recall % (Validation)	Macro F1 score % (Validation)	Macro precision % (Test)	Macro recall % (Test)	Macro F1 score % (Test)
Tree	Medium Tree	78.41	78.72	80.16	78.41	78.26	81.12	78.72	78.33
	Coarse Tree	49.16	49.95	52.54	49.16	48.55	51.67	49.95	48.66
Naive Bayes	Gaussian Naive Bayes	38.20	38.46	42.33	38.20	34.41	42.10	38.46	34.72
	Kernel Naive Bayes	69.09	71.66	72.42	69.09	68.95	74.51	71.66	71.63
SVM	Linear SVM	71.59	72.01	71.60	71.59	71.52	72.07	72.01	71.96
	Coarse Gaussian SVM	71.16	74.05	71.13	71.16	71.13	74.01	74.05	74.02
Ensemble	Subspace Discriminant	64.25	64.05	64.21	64.25	64.21	63.96	64.05	63.97
	Subspace KNN	68.36	73.70	68.47	68.36	68.41	73.69	73.70	73.69
	RUSBoosted Trees	78.41	78.72	80.16	78.41	78.26	81.12	78.72	78.33

An examination of Table 12 reveals that the obtained results range from 97% to 99.9%. Among decision trees, Fine Tree, among support vector machines, Cubic SVM, among nearest neighbor algorithms, Fine KNN, among kernel algorithms, SVM Kernel, among ensemble trees, Bagged Tree, and among neural networks, Narrow Neural Network produced the best results. The high performance observed across different models suggests that the fused feature representation contains useful information for the classification task within the adopted experimental setting. Various studies have shown that data fusion improves performance compared to using a single data source^102–104. Although the literature suggests that fusion improves performance, in the present study, controlled laser-only and acoustic-only baseline experiments were not included in the same evaluation protocol, therefore, it cannot be said that there was a specific performance increase attributable to fusion. The worst results obtained in the study are shown in Table 13.

The results in Table 13 show the results of the low performance obtained. Among these models, the worst performance was achieved by the Coarse Tree among decision trees, the Gaussian Naive Bayes among Naive Bayes models, the Coarse Gaussian SVM among support vector machines, and the Subspace Discriminant model among ensemble models. The lowest performance range was observed to be between 38 and 78%.

The training and testing times for each model are given in Table 14. Table 14 also includes the hyperparameters of the models used. The model size is also given in bytes to facilitate analysis and comparison of the resource requirements of each model.

Table 14.

Hyperparameters, training and testing times, and model size of the model used.

	Preset	Prediction speed (obs/sec)	Training time (sec)	Model size (mb)	Hyperparameters
Tree	Fine Tree	199,460.37	8.06	0.03	Maximum number of splits: 100; Split criterion: Gini’s diversity index; Surrogate decision splits: Off
	Medium Tree	215,682.95	6.50	0.02	Maximum number of splits: 20; Split criterion: Gini’s diversity index; Surrogate decision splits: Off
	Coarse Tree	201,316.36	4.24	0.01	Maximum number of splits: 4; Split criterion: Gini’s diversity index; Surrogate decision splits: Off
Naive Bayes	Gaussian Naive Bayes	116,141.60	9.87	0.04	Distribution name for numeric predictors: Gaussian; Distribution name for categorical predictors: Not Applicable
	Kernel Naive Bayes	65.91	2,186.66	44.61	Distribution name for numeric predictors: Kernel; Distribution name for categorical predictors: Not Applicable; Kernel type: Gaussian; Support: Unbounded; Standardize data: Yes
SVM	Linear SVM	136,027.65	716.03	0.07	Kernel function: Linear; Kernel scale: Automatic; Box constraint level: 1; Multiclass coding: One-vs-One; Standardize data: Yes
	Quadratic SVM	46,834.80	476.87	0.85	Kernel function: Quadratic; Kernel scale: Automatic; Box constraint level: 1; Multiclass coding: One-vs-One; Standardize data: Yes
	Cubic SVM	105,060.76	228.82	0.39	Kernel function: Cubic; Kernel scale: Automatic; Box constraint level: 1; Multiclass coding: One-vs-One; Standardize data: Yes
	Fine Gaussian SVM	7,101.95	1,857.39	1.70	Kernel function: Gaussian; Kernel scale: 1.6; Box constraint level: 1; Multiclass coding: One-vs-One; Standardize data: Yes
	Medium Gaussian SVM	2,550.18	463.74	4.77	Kernel function: Gaussian; Kernel scale: 6.5; Box constraint level: 1; Multiclass coding: One-vs-One; Standardize data: Yes
	Coarse Gaussian SVM	802.98	2,022.63	18.11	Kernel function: Gaussian; Kernel scale: 26; Box constraint level: 1; Multiclass coding: One-vs-One; Standardize data: Yes
KNN	Fine KNN	1,444.46	2,020.55	11.87	Number of neighbors: 1; Distance metric: Euclidean; Distance weight: Equal; Standardize data: Yes
	Medium KNN	1,107.97	2,019.08	11.87	Number of neighbors: 10; Distance metric: Euclidean; Distance weight: Equal; Standardize data: Yes
	Coarse KNN	1,331.39	2,018.03	11.87	Number of neighbors: 100; Distance metric: Euclidean; Distance weight: Equal; Standardize data: Yes
	Cosine KNN	1,409.26	2,016.72	11.87	Number of neighbors: 10; Distance metric: Cosine; Distance weight: Equal; Standardize data: Yes
	Cubic KNN	225.73	2,015.70	11.87	Number of neighbors: 10; Distance metric: Minkowski (cubic); Distance weight: Equal; Standardize data: Yes
	Weighted KNN	1,106.19	2,014.63	11.87	Number of neighbors: 10; Distance metric: Euclidean; Distance weight: Squared inverse; Standardize data: Yes
Kernel	SVM Kernel	2,679.67	1,822.93	0.18	Learner:SVM; Number of expansion dimensions:Auto; Regularization strength (Lambda): Auto; Kernel scale: Auto; Multiclass coding: One-vs-One; Standardize data: Yes; Iteration limit: 1,000
	Logistic regression Kernel	2,507.49	1,630.32	0.18	Learner:Logistic Regression;Number of expansion dimensions:Auto; Regularization strength (Lambda):Auto; Kernel scale:Auto; Multiclass coding:One-vs-One; Standardize data:Yes; Iteration limit:1,000
Ensemble	Boosted Trees	31,365.62	1,627.50	0.46	Ensemble method: AdaBoost; Learner type: Decision tree; Maximum number of splits: 20; Number of learners: 30; Learning rate: 0.1; Number of predictors to sample: Select All
	Bagged Trees	35,573.27	1,626.26	0.91	Ensemble method: Bag; Learner type: Decision tree; Maximum number of splits: 34,559; Number of learners: 30; Number of predictors to sample: Select All
	Subspace Discriminant	7,054.04	1,371.97	0.46	Ensemble method: Subspace; Learner type: Discriminant; Number of learners: 30; Subspace dimension: 21
	Subspace KNN	627.05	2,508.81	190.03	Ensemble method: Subspace; Learner type: Nearest neighbors; Number of learners: 30; Subspace dimension: 21
	RUSBoosted Trees	44,593.85	98.08	0.46	Ensemble method: RUSBoost; Learner type: Decision tree; Maximum number of splits: 20; Number of learners: 30; Learning rate: 0.1; Number of predictors to sample: Select All
NN	Narrow Neural Network	146,509.71	256.19	0.01	Number of fully connected layers: 1; First layer size: 10; Activation: Rectified Linear Unit (ReLU); Iteration limit: 1,000; Regularization strength (Lambda): 0; Standardize data: Yes

Table 14 provides basic information regarding the training and testing times for the models. This information also provides basic information regarding the hardware requirements for using the models. The values provided here can be used to determine which model to choose for real-time systems and systems with low hardware requirements. The hyperparameter information used in each model during training is also included in the table.

The results of studies on detecting human orientation outside the field of view have been tested with various artificial intelligence and machine learning methods. The results indicate that the fused feature set is effective for orientation classification. Features derived from the acoustic chirp signal and the laser enhance representational richness. The observed performance demonstrates that the fused feature set is informative under the evaluated experimental conditions. When the results are examined, for each model:

1. Decision trees: Fine Tree achieved higher performance compared to other decision trees. A result of 99.92% was higher than those of the other decision tree variants. In contrast, Medium Tree and Coarse Tree exhibited significantly lower performance. Medium Tree achieved a very low performance of 78%, while Coarse Tree achieved a very low performance of 49%. This suggests that this is due to the number of nodes and depth in the decision trees. It can be said that simplified tree structures lose their generalization ability.

2. Support vector machines: Many SVM-based models demonstrated high performance. Cubic SVM produced an F1 score of 99.8%, Quadratic SVM 99.7%, Fine Gaussian SVM 99.6%, and Medium Gaussian SVM 98.6%. In contrast, Linear SVM had low F1 scores of 72% and Coarse Gaussian SVM 74%.

3. K-Nearest neighbor models: Fine KNN, Weighted KNN, Cosine KNN, and Medium KNN models demonstrated f1 score performance above 99%. Coarse KNN, at 98.6%, performed relatively lower than the other models. This demonstrates the importance of the number of neighbors and distance metric parameters for this situation.

4. Kernel and logistic regression: Kernel-based models appear to accurately separate the nonlinear dataset. The SVM Kernel achieved high performance at 99.6%, and the Logistic Regression Kernel at 99.3%.

5. Naive bayes: These were the lowest-performing models among the models used in the study. Gaussian Naive Bayes achieved 34.7% accuracy, and Kernel Naive Bayes achieved 71.6% accuracy. The low performance is believed to be due to the lack of unconditional independence of the features extracted from laser and acoustic chirp signal.

6. Ensemble models: Ensemble models achieved the highest performance among the models used. Boosted Trees and Bagged Trees achieved high performance at 99.9%. However, Subspace Discriminant achieved low performance at 64%, Subspace KNN at 73.7%, and RUSBoosted Trees at 78.3%. This may indicate that some ensemble models exhibit generalization problems and overfit to complex data distributions.

7. Neural network: This model is among the models that produce the highest f1 scores. The Narrow Neural Network achieved a high performance of 99.8%. This suggests that artificial neural networks can effectively model data consisting of laser and acoustic chirp signal.

8. LAO-Net: The feedforward artificial neural network is the model with the highest performance in the study. The model’s 99.98% score indicates strong performance on the collected dataset under the adopted evaluation protocol. Nevertheless, this result should be interpreted cautiously because the dataset was collected from eight subjects in a controlled laboratory environment. The model learned features resulting from the fusion of laser and acoustic chirp signal. It demonstrated very strong performance on the dataset collected under controlled experimental conditions.

The confusion matrix obtained from the testing process of each model used in the study was used to estimate the human’s estimation process for the models facing behind, front, left and right. This confusion matrix is presented as a separate metric that reveals the performance of the models. The confusion matrices of the models that produced the best results are shown in Fig. 13.

Fig. 13.

Confusion Matrix of the best performing models.

Providing a confusion matrix for the models with the lowest performance, in addition to the models with the highest performance, would be valuable for comparative analysis. The confusion matrices for the models with the lowest performance in the application are shown in Fig. 14.

Fig. 14.

Confusion Matrix of the lowest performing models.

Correlation and PCA-based feature analysis

To examine the fused 21 + 21 feature set derived from laser and acoustic data in greater detail, pairwise correlation analysis and PCA (Principal Component Analysis) were performed. The extracted feature set includes statistical, temporal, amplitude-based, and spectral descriptors. Therefore, there may be redundancy among these descriptors. Therefore, a correlation matrix was used to determine the bivariate relationships between the features. PCA-based variance analysis was used to see how the total variance of these features is distributed among the principal components. The analyses were performed to provide clear information regarding the redundancy or complementarity of the features.

The correlation matrix shown in Fig. 15 reveals moderate to high levels of correlation between some of the laser and acoustic properties. This indicates a strong correlation among them. The various features used for each data source in this study were designed to characterize similar aspects of the signal, such as amplitude variation, energy content, and distributional characteristics. At the same time, the correlation structure indicates that not all features are strongly correlated. This suggests that the fused feature set also preserves complementary information from different descriptor groups. Therefore, redundancies are observed among the features. However, the feature set appears to capture multiple distinct characteristics of the laser-acoustic signals. For visual clarity, the strongest relationships in Fig. 15 are interpreted at the group level rather than on a feature-by-feature basis.

Fig. 15.

Pairwise correlation matrix of the extracted fused features used in the classification task.

The PCA-based explained variance analysis provides additional insight into the structure of the fused feature space. The distribution of variance across the principal components indicates how the feature space should be represented. The variance is not concentrated in one or a few components. This indicates that the information in the feature set is distributed across multiple dimensions. Some redundancies are observed in the features present in the dataset used in this study. This supports the interpretation of feature contents that possess discriminative potential for the combined classification task. Figure 16 presents the PCA-based explained variance distribution of the fused feature set.

Fig. 16.

Explained variance distribution of the principal components obtained from the fused feature set.

Discussion

The data obtained in this study was obtained using an experimental environment. Efforts were made to replicate the real-world environment. This study aimed to detect a person’s orientation in a NLOS environment. Laser and acoustic chirp signal data sources were used for detection. Data from both sources were then extracted, and these features were combined into a single dataset using data fusion. This created a fused dataset for detecting human orientation in NLOS environments. It appears that the characteristics of both data sources contribute to the observed model performance.

The acoustic linear chirp signal’s broadband energy impinges on walls and objects outside the field of view, creating a secondary reflection. This secondary reflection provides a distinctive feature. MFCC/STFT-based features capture statistical differences. They provide information regarding human presence (presence/absence) and human orientation. In addition to this data source, the laser has been shown to produce supporting data. Low photon fluctuations resulting from wall reflections of the signal sent from the laser source pose a problem. The efficiency of the laser data was increased by preprocessing the laser data obtained from the experimental environment, i.e., the white noise, obtained from an empty environment, by subtracting it from the real data. Laser and acoustic chirp data are considered complementary features in the experimental setup. In this study, the combined laser-acoustic representation achieved strong performance for human orientation classification. However, since this study does not include a controlled comparison between laser-only, acoustic-only, and fused representations under the same evaluation protocol, the specific incremental contribution of feature fusion cannot be quantified directly and should therefore be interpreted with caution. Literature searches have focused on human presence/absence detection in NLOS environments. These studies generally emphasize success rates exceeding 90%. This study focused on the more specific task of orientation classification and achieved 99.98% accuracy under the experimental conditions evaluated. The results should not be interpreted as direct proof of generalizability to more complex real-world NLOS environments. Specifically, the current findings were obtained under a controlled setup involving a single reflective wall, limited environmental variability, and a small pool of participants. This limits outdoor validity. In addition, because multiple window-level samples were derived from the same original laser scan, the reported results should not be interpreted as if they were obtained from a set of fully independent observations.

An examination of the results reveals that different features are effective in detecting the orientation of people outside the models’ field of view. Some features are more effective in detecting people with their behind turned, while others are more effective when people are facing forward. Figure 17 shows which features are more influential in determining human orientation. Class 1 represents the effective features for people facing behind, Class 2 represents the effective features for people facing front, Class 3 represents the effective features for people facing left, and Class 4 represents the effective features for people facing right.

Fig. 17.

Global Shapley-based feature importance plot for the fused laser–acoustic classifier.

The performance of the models depends on the accuracy of the dataset and the model’s performance. The features in the dataset emerge as the discriminatory power in the model during training. Figure 17 more clearly demonstrates the features and their effectiveness. The figure shows that the most effective feature in Classes 1 and 3 is L_feat_dom_power, derived from the laser data. However, the second most effective feature in these two classes differs. The overall dominance reveals that some features are more significant. In Classes 2 and 3, the determining feature is A_feat_spec_entropy, which belongs to the acoustic signal. These features are found to be more decisive when determining whether a person is facing behind or right.

The most decisive attributes and the graph of the effectiveness of these attributes in determining the orientation in which a person is are given in Fig. 18.

Fig. 18.

Mean absolute Shapley values of the most influential fused features for orientation classification.

Figure 18 shows which attribute is most effective in determining human orientation. The degree of effectiveness in human orientation varies depending on the length of the bar. This feature, brought about by explainable artificial intelligence, shows that the L_feat_dom_power attribute from the laser data is the most powerful attribute in determining orientation. The second most influential attribute was A_feat_spec_entropy from the acoustic chirp signal data.

An examination of the effectiveness of features in orientation finding reveals information regarding the effectiveness of both laser and audio data in determining orientation. This finding suggests that the dataset created by combining laser and audio features contains useful information for orientation determination. These results suggest that the early fusion process may have contributed positively to the observed classification performance.

Correlation matrix and PCA-based analyses reveal the relationships among certain variables. Furthermore, they have demonstrated that the fusion feature set preserves complementary information across multiple dimensions.

The very high accuracies (> 99%) obtained by LAO-Net and some ML models raise the possibility of overfitting. The lack of sensor-related data variation and recording the dataset under similar conditions reinforce the possibility of overfitting. However, the results of different models indicate that performance may be low. Although fivefold cross-validation and an independent test split were used, these procedures do not fully eliminate the limitations arising from the controlled setup, the limited number of subjects, and the absence of subject-independent evaluation. This evaluation framework helps provide more stable performance predictions. Therefore, further validation under stricter, subject-independent protocols is still necessary. Furthermore, the results show that the data fusion performs well. The accuracy difference between training and validation is quite low. This small difference may indicate a limited distinction between training and validation performance. However, a more extensive validation is still required to assess its generalizability. Similarly, the closeness of the F1 scores between the training and validation sets may indicate a limited tendency toward overfitting in the current dataset.

It is believed that the high matrix complexity performance may be due to the geometry of the controlled experimental environment created in this study. The fixed positions of the wall, the participant, the laser source, and the acoustic setup may have produced a dependent signal pattern. However, these results should be interpreted within the context of the controlled environment used in this study, and further validation is required in more complex real-world NLOS environments.

Limitations and future work

Despite the success of the study, it also has some limitations. The dataset was obtained from eight subjects and conducted in a controlled laboratory environment. This may limit the ability to generalize to the acoustic reflections of people with different physical characteristics. Furthermore, although the data obtained from the devices installed in the experimental environment is reliable, re-acquiring data with different devices may cause performance degradation due to calibration differences. Using different reflective surfaces in the experimental environment, rather than the wall designated as the reflective surface, could affect performance and stability. To overcome this, future studies will attempt to create datasets that include different reflective surfaces.

One methodological limitation of the study concerns the use of multiple 36-sample windows derived from the same laser scan. To capture local temporal variations, multiple 36-sample segments were obtained from the same 16,220-line laser recording. Although this strategy increased the level of detail in feature extraction, segments derived from the same scan may retain some correlation. Therefore, rather than being interpreted as a set of completely independent observations for the dataset, it should be understood as a window-based representation of the recorded signals. This limitation necessitates future studies to investigate record-level and subject-independent validation strategies to better assess robustness under stricter independence assumptions.

The study did not employ a LOSO (leave-one-subject-out) method. This is considered another limitation of the study. The dataset comprises the complete set of records collected within a fixed experimental setup. The dataset was treated as a whole. Training and test data were randomly selected from this dataset. A subject-independent validation process involving a larger dataset and a broader pool of participants represents an important direction for future studies.

The experimental setup utilized a single reflective wall under controlled acoustic conditions. Real-world off-line-of-sight environments may include heterogeneous wall materials, varying wall thicknesses, cluttered furniture, and ambient noise. A single reflective surface can influence signal behavior and classification performance. Therefore, the findings of this study should be interpreted within the controlled laboratory environment adopted here. Future studies plan to incorporate multiple reflective surfaces.

Another limitation of this study is the absence of a controlled comparison between laser-only, acoustic-only, and fused representations under the same evaluation protocol. Therefore, although the fused dataset demonstrated strong performance under the adopted experimental conditions, the exact performance gain attributable to fusion could not be quantified in the present study. A systematic baseline comparison between single-modality and fused inputs remains an important direction for future research.

The study was conducted with eight subjects. Furthermore, the fact that data collection was conducted in a controlled experimental environment and that a wall was used as a reflective surface are some limitations of the study. In the future, datasets that include different reflective surfaces, a larger number of subjects, and a wider variety of noise conditions may help mitigate these limitations.

One of the limitations of this study is that it was conducted solely on a dataset we created ourselves. The primary reason for this, to the best of our knowledge, is the lack of a publicly available dataset for NLOS laser–acoustic human orientation classification. This study focuses on laser-acoustic sensing; in future studies, a more comprehensive validation study should be conducted once a similar publicly available dataset becomes available.

Conclusion

This study aimed to determine human orientation in an NLOS environment by using data fusion of laser and acoustic chirp signals. Twenty-one features were determined from each data source. A dataset with a total of 42 features was created: 21 features from the laser signal and 21 features from the acoustic chirp signal. This dataset was combined using the early fusion method. The fused dataset yielded strong performance across various machine learning models as well as the proposed feedforward neural network model under the adopted experimental protocol. The results suggest that the combined feature representation is effective across multiple evaluated models.

LAO-Net achieved 99.98% accuracy on the collected dataset under the adopted evaluation protocol. However, this result reflects performance within a controlled laboratory setting involving eight subjects and should therefore be interpreted cautiously. Some of the other ML/AI models used in the study achieved success rates above 99%. Bagged Trees, Fine KNN, and Cubic SVM are some of these models. An evaluation of the findings revealed that the fused laser-acoustic representation achieved strong classification performance under the experimental protocol. The fused feature representation was associated with stable and consistently strong model performance under the tested conditions. Nevertheless, because single-modality baselines were not evaluated under the same protocol, the specific benefit contributed by feature fusion could not be quantified directly in the present study and remains to be validated in future work. Furthermore, a global Shapley analysis was conducted, and the features provided discriminative information. Global Shapley analysis identified L_feat_dom_power and A_feat_spec_entropy as the most influential features for orientation prediction. Performance should be interpreted within the framework of the adopted window-based experimental protocol. Feature importance analysis was performed based on the global Shapley values obtained using the MATLAB Classification Learner. The results were summarized using the average absolute Shapley values across all analyzed observations.

The study utilized numerous ML/AI models. Each model, both performing well and performing poorly, is presented comparatively in tables. This may provide a useful reference for researchers conducting similar studies. In addition to its academic contributions, this study may hold potential significance for applications such as search and rescue, security systems, autonomous platforms, and robotic sensing. This study holds potential for post-disaster assessment and related decision-support scenarios. This is an important step towards detecting human orientation in out-of-line-of-sight environments under controlled experimental conditions. However, broader validation under more realistic NLOS conditions, including different wall materials, greater environmental variability, and a larger participant pool, is still required to establish external validity.

Future studies aim to create a new dataset that incorporates different surface materials, a larger number of subjects, and complex environmental conditions. Furthermore, the resulting dataset will be studied with deep learning models, hybrid models, and transformer-based methods. A larger dataset is planned for future studies. In addition, subject-independent validation protocols, such as LOSO evaluation, will be considered in future experiments.

Author contributions

The entire study was organized by a single author.

Funding

Türkiye Bilimsel ve Teknolojik Araştırma Kurumu, 123E386.

Data availability

The data supporting the findings of this study were obtained from project number 123E386, supported by TUBITAK. The data cannot be publicly shared at this time due to the ongoing project and patent process. However, the data may be made available from the author upon reasonable request, subject to project, intellectual property, and patent-related restrictions. To support methodological reproducibility despite current data-sharing restrictions, supplementary pseudocode describing the preprocessing, feature extraction, fusion, and evaluation workflow has been provided.

Declarations

Competing interests

The authors declare no competing interests.