Task-oriented visual SLAM: a comprehensive map classification framework for dynamic indoor robot manipulation

Abstract

Visual Simultaneous Localization and Mapping (vSLAM) is fundamental to enabling robotic mobile manipulation—i.e., the seamless integration of navigation, perception, and dexterous interaction with objects in unstructured environments. Yet current vSLAM research largely lacks a principled, task-oriented framework for map classification, resulting in suboptimal map representations that hinder robustness and efficiency in dynamic indoor settings. To bridge this gap, we propose a purpose-driven taxonomy of vSLAM maps specifically designed for mobile manipulation tasks. This taxonomy comprises four complementary categories: geometric 3D maps, semantic maps, object-level maps, and hybrid maps—each distinguished by its representational granularity, functional scope, and suitability for downstream manipulation primitives. We provide a systematic comparative analysis of their construction pipelines, underlying technical assumptions, and real-world deployment contexts, evaluating them rigorously across three critical dimensions: environmental adaptability, pose estimation accuracy, and real-time computational feasibility. Finally, we synthesize key limitations in existing approaches and identify concrete, high-impact directions for future work—including tight coupling between mapping semantics and manipulation affordances, and scalable learning-based map fusion.

Introduction

The mobility of robots underpins their ability to perform daily tasks. Enabling autonomous execution and self-control in mobile manipulation fundamentally relies on the construction of an accurate environmental model. Different types of environmental models serve distinct roles in supporting mobile manipulation, as illustrated in Fig. 1. Selecting appropriate technologies to construct effective environmental models is therefore essential for successful robotic manipulation.

Fig. 1.

Roles of different types of maps in supporting mobile manipulation. (a) 3D maps provide an intuitive representation of the spatial structure of the environment. (b) Semantic maps associate objects with relevant semantic information. (c) Object-level maps capture detailed properties and states of individual objects. (d) These maps collectively guide robotic task execution, highlighting their distinct roles from environmental modeling to actual manipulation^1–4.

Currently, visual sensors, LiDAR, ultrasound, and inertial navigation are widely employed to perceive the environment and establish models. Each traditional sensing modality, however, presents inherent limitations. For instance, ultrasound suffers from limited range and resolution, while inertial navigation accumulates drift over time. LiDAR provides precise depth information and maintains robustness under extreme lighting conditions or in texture-less environments, but it offers limited semantic understanding. In contrast, visual sensors excel in dynamic indoor environments, particularly in semantic perception and texture-rich scenarios, though they may struggle in low-texture or large-scale scenes. Consequently, multi-sensor fusion approaches, such as visual-inertial or visual-LiDAR systems, have garnered increasing attention. Among these technologies, visual SLAM (vSLAM) has emerged as a prominent research focus due to its ability to balance perceptual richness and real-time performance using low-cost cameras. SLAM technology is particularly critical for addressing challenges in dynamic indoor environments, such as fluctuating lighting, complex spatial structures with numerous moving objects, and substantial variations in object scale. By enabling real-time perception of environmental changes, efficient processing of dynamic objects, and accurate map construction, vSLAM has become a central topic in mobile robotics research.

A large body of research has explored and reviewed vSLAM methods. Early approaches primarily relied on feature-point-based techniques. Among these, Oriented FAST and Rotated BRIEF SLAM (ORB-SLAM) is a representative method that employs ORB features for image matching and achieves rotation invariance in dynamic environments. The ORB feature-matching results for a sample image are shown in Fig. 2. By performing multi-scale image analysis, ORB-SLAM improves feature extraction efficiency. Qing et al.⁶ provided a chronological survey of milestone methods and the standard vSLAM pipeline, highlighting the advantages of combining point and line features. Kazerouni et al.⁵ classified SLAM methods based on feature types and summarized existing vSLAM approaches. Despite their success, feature-point methods are limited by computational demands and difficulties in handling complex scenes, motivating the development of direct methods. Chen et al.⁷ reviewed both feature-based and direct visual odometry methods, detailing their implementation and providing a comparative analysis. However, direct methods remain sensitive to lighting variations, which limits their practical applicability.

Fig. 2.

The ORB image matching results for the sample image⁵.

With the rise of deep learning, researchers have integrated neural networks into vSLAM. Li et al.⁸ compiled state-of-the-art end-to-end deep learning SLAM approaches, combining semantic segmentation with emerging semantic SLAM techniques to achieve simultaneous ego-motion estimation and high-level scene understanding. Li et al.⁹ examined the application of deep learning in vSLAM from four neural network-based perspectives, categorizing existing studies according to their characteristics, strengths, and limitations, and discussing potential future directions. Arshad et al.¹⁰ proposed a taxonomy for state-of-the-art deep learning-based loop closure detection algorithms, providing detailed comparative metrics and a comprehensive review of deep learning-based methods.

Existing surveys on vSLAM have offered valuable insights from multiple perspectives. Some categorize methods according to algorithmic paradigms, such as feature-based versus direct approaches^5,7 or deep learning-based techniques^8–10, while others focus on sensor configurations or application domains¹¹. Although these classifications effectively capture the evolution of SLAM algorithms, they do not directly address a question central to robotic manipulation: which type of map should a robot construct to optimally support a given task?

To address this gap, we propose a task-oriented map taxonomy that organizes existing methods according to the type of output map—3D, semantic, object-level, and hybrid—rather than the underlying algorithms. Unlike prior classifications that emphasize how maps are constructed, our framework emphasizes what maps are generated and the specific tasks they enable. This distinction is particularly important for manipulation applications, where the choice of map representation directly affects task feasibility and efficiency. We further refine our discussion to dynamic indoor environments, where moving objects and changing layouts impose additional requirements on map representation.

As summarized in Table 1, existing classifications are largely method-centric, organizing works by algorithmic implementation, including feature extraction, optimization strategies, or learning architectures. In contrast, our taxonomy is task-centric, classifying maps by their representational output (3D, semantic, object-level, hybrid) and explicitly linking each map type to the manipulation tasks it facilitates.

Table 1.

Comparison between existing SLAM surveys and the proposed task-centric taxonomy.

Aspect	Existing surveys (e.g. 5–10)	Our taxonomy
Classification criterion	Algorithmic paradigm	Map representation type
Perspective	Method-centric (“How is the map built?")	Task-centric ("what map enables which task?")
Metric focus	SLAM metrics (accuracy, efficiency)	Task-level metrics (success rate, completion rate)

Based on the principles of SLAM, this paper proposes a map classification framework for mobile manipulation. The framework systematically analyzes and compares the advantages and limitations of different map types required to support robotic mobile manipulation, thereby providing a structured knowledge reference and identifying new directions for research and practical applications in this field. The main contributions of this paper are as follows:

1. We propose a dynamic indoor vSLAM map classification framework for mobile manipulation. Maps are clearly categorized into four types: 3D, semantic, object-level, and hybrid, addressing a gap in existing classifications from a task-centric perspective.

2. The construction methods, technical characteristics, and applicable scenarios of these four map types in dynamic indoor environments are systematically organized. Comparative analyses are provided based on key metrics such as dynamic adaptability, localization accuracy, and computational efficiency, offering a basis for selecting appropriate map types according to specific task requirements.

3. Leveraging the classification framework, we identify current challenges—including real-time performance, occlusion handling, and semantic consistency—and highlight future research opportunities, such as lightweight models, online map updates, and cross-modal fusion.

The remainder of this paper is organized as follows. “Overview of vSLAM” section reviews the fundamental principles, basic processes, and existing theories of vSLAM. “3D maps” through “Hybrid maps” sections provide detailed discussions of 3D maps, semantic maps, object-level maps, and hybrid maps, respectively. Finally, “Summary of trade-offs across map types” section summarizes the work and outlines future research directions.

Survey methodology

The literature reviewed in this paper was identified through a multi-stage search strategy using the Web of Science Core Collection. Rather than relying on a fixed set of keywords, an iterative search process was employed. Initial broad searches were refined based on keywords extracted from relevant papers, and new search terms were added as additional map types or methods emerged during the review. The search keywords included:

1. SLAM technologies: (“visual SLAM” OR “vSLAM” OR “simultaneous localization and mapping”);

2. 3D maps: (“octomap” OR “NeRF” OR “3D Gaussian splatting” OR “point cloud” OR “occupancy map” OR “TSDF” OR “ESDF”);

3. Semantic maps: (“semantic map” OR “semantic segmentation” OR “Mask R-CNN” OR “YOLO” OR “semantic SLAM” OR “DS-SLAM”);

4. Object-level maps: (“object-level map” OR “instance segmentation” OR “object SLAM” OR “MaskFusion” OR “OA-SLAM”);

5. Hybrid maps: (“hybrid map” OR “hierarchical SLAM” OR “multimodal fusion” OR “topological map” OR “metric map” OR “Kimera” OR “ESLAM”);

6. Application context: (“dynamic environment” OR “indoor” OR “mobile manipulation” OR “grasping” OR “pick-and-place” OR “navigation”).

The search was restricted to English-language, peer-reviewed papers published up to April 2026. Inclusion criteria required studies to present vSLAM methods for map construction in indoor or dynamic environments with experimental validation. Exclusion criteria included non-English publications, works lacking a clear methodology, purely sensor-focused hardware studies, and studies exclusively addressing 2D mapping.

The screening process comprised title and abstract screening, followed by full-text review, and backward snowball sampling from the references of included papers and existing vSLAM surveys^5–10. The final reference list includes 150 papers, as detailed in the References section.

Overview of vSLAM

Basic principles of vSLAM

SLAM is a technique that estimates sensor motion while reconstructing the structure of unknown environments. When implemented using cameras, it is referred to as visual SLAM (vSLAM). This approach relies solely on camera data to simultaneously generate a map of the environment and perform self-localization. The accuracy and quality of the map produced by SLAM are critical, as they directly influence the performance of subsequent autonomous functions, including positioning, path planning, and obstacle avoidance.

Basic process of vSLAM

vSLAM relies on sequential image data captured by a camera to reconstruct the spatial relationships between the environment and the camera itself. By exploiting image information and camera imaging models, vSLAM incrementally generates a map of the surrounding environment while simultaneously estimating the camera’s position within it. The general processing workflow of vSLAM can be divided into three stages: front-end processing, back-end optimization, and loop-closure verification¹¹, as illustrated in Fig. 3. The front end, often referred to as the visual odometry module, processes image data to estimate the camera’s pose at each time step, thereby forming the trajectory of the camera. The back end refines these pose estimates to minimize accumulated errors and complete the construction of the environmental map. In frameworks that incorporate loop closure detection, images captured by the camera are used to identify previously visited locations, and the resulting information is fed back to the back-end optimization for further refinement of both the trajectory and the map.

Fig. 3.

Basic Flowchart of vSLAM¹¹.

Existing theories of vSLAM

vSLAM can be broadly categorized into three main approaches. The first is the feature-based method, which completes the SLAM process by detecting and matching feature points in the images. This approach is computationally efficient and performs well in environments with rich textures. The second approach is the direct method, which bypasses feature extraction and directly utilizes pixel intensity information for SLAM. By leveraging all available image information rather than only feature points, the direct method is effective even in low-texture environments. However, it entails a higher computational burden and demands greater computing resources. The third approach is based on deep learning, employing neural networks to extract semantic information and features for pose estimation and map construction. This approach can automatically learn robust representations, exhibits strong scene understanding capabilities, and demonstrates enhanced robustness and adaptability, particularly in dynamic indoor environments.

During mobile manipulation, a robot’s localization typically relies on the construction of an environmental map. By continuously sensing the surroundings and updating the map, the robot determines exploration directions and action strategies, thereby enabling comprehensive coverage of the environment. Based on visual SLAM (vSLAM) technology, we classify the maps required to support robot mobility in dynamic indoor environments into four categories according to their representation, as shown in Fig. 4: three-dimensional (3D) maps, semantic maps, object-level maps, and hybrid maps. Each category is described in detail below.

Fig. 4.

Map types for mobile manipulation: Four map types—3D, semantic, object-level, and hybrid—are illustrated, with their technical foundations, construction methods, advantages, and limitations summarized. This classification provides a concise framework for understanding map construction and its applications in mobile manipulation.

The proposed four-category taxonomy is derived from two orthogonal dimensions that reflect the functional requirements of manipulation tasks, as synthesized from the literature reviewed in “3D maps” to “Hybrid maps” sections. The first dimension is the level of abstraction, ranging from raw geometric data to semantic understanding and finally to individual object identity:

1. 3D maps capture the environment at the geometric level, representing metric structure, shape, and spatial relationships. This level suffices for navigation and obstacle avoidance but lacks object-level distinction.

2. Semantic maps augment geometric data with categorical labels (e.g., “chair,” “table”) and scene functions, supporting high-level reasoning such as task-oriented navigation and human–robot interaction.

3. Object-level maps further refine semantic information to individual instances, providing unique object identities, dynamic attributes, and behavioral states. This level is essential for manipulation tasks requiring interaction with specific objects, such as grasping or fetching.

The second dimension is the degree of integration, spanning from single-modality representations to multimodal fusion:

1. 3D, semantic, and object-level maps can be generated using either a single sensor modality or fused data from multiple sources.

2. Hybrid maps represent the highest integration level, combining multiple representation types or abstraction levels within a unified framework. This enables robots to achieve both geometric precision for navigation and semantic understanding for task planning.

Together, these two dimensions define a design space for task-oriented mapping. Specific manipulation tasks may demand particular combinations of abstraction and integration; for instance, grasping requires object-level abstraction with geometric accuracy, whereas long-term operation in dynamic indoor environments benefits from hybrid maps to maintain global consistency while updating local changes. This principled, task-oriented basis distinguishes our taxonomy from existing classifications, which primarily organize methods by algorithmic implementation^5,7–10 rather than by representational suitability for tasks.

Three key differences distinguish our taxonomy from prior classifications. First, existing surveys categorize SLAM methods by algorithmic approach (feature-based, direct, or deep learning), reflecting how sensor data are processed. In contrast, our taxonomy categorizes maps by representation type (3D, semantic, object-level, hybrid), reflecting the nature of the produced output. Second, existing classifications are method-centric, emphasizing algorithmic performance metrics such as accuracy and efficiency, whereas our taxonomy is task-centric, explicitly linking map types to the manipulation tasks they enable (e.g., object-level maps for grasping, hybrid maps for navigation in cluttered environments). Third, prior studies often treat map construction as the endpoint of SLAM, whereas our framework positions maps as a means to an end—a core enabler of the robot’s mobile manipulation loop. Consequently, our taxonomy is complementary to, rather than a replacement of, existing classifications.

To demonstrate the practical value of the proposed framework, we establish a mapping between typical mobile manipulation tasks and the optimal map types recommended by this taxonomy, as shown in Fig. 5. This mapping is derived from the functional requirements of each task: Obstacle-avoidance navigation prioritizes geometric accuracy (3D maps); specific object grasping requires precise object pose estimation (object-level maps); human–robot collaborative interaction benefits from semantic understanding (semantic maps); and pick-and-place in complex scenes or long-term operation in dynamic environments requires integrated representations (hybrid maps). This hierarchical task–map mapping provides intuitive validation, demonstrating how the classification framework informs map selection in real-world scenarios—a perspective largely absent in methodology-centric reviews.

Fig. 5.

Mapping between typical robotic manipulation tasks and the optimal map types recommended by the proposed classification framework.

Based on this task-oriented taxonomy, we subsequently examine each map type in detail, discussing construction methods, technical characteristics, and applicability to mobile manipulation.

3D maps

In dynamic indoor environments, mobile robot manipulation faces a range of complex challenges. Frequent human movement, rearrangement of facilities, multi-floor structures, and dynamic elements such as temporary obstacles impose stringent requirements on real-time performance, positional accuracy, and three-dimensional spatial perception and management. 3D maps, with their capabilities for stereoscopic modeling, real-time updating, and multi-source data fusion, are well-suited to address these challenges.

A 3D map represents geographic spatial information in three dimensions, providing a realistic depiction of terrain, buildings, and objects. In dynamic indoor settings, 3D maps capture structural details, object locations, and shapes, enabling robots to achieve comprehensive and in-depth perception of their surroundings. This enhanced spatial understanding forms a critical foundation for subsequent mobile manipulation tasks.

Technical support for 3D maps

3D maps are grounded in the principles of three-dimensional (3D) reconstruction technology. 3D reconstruction refers to the process of generating a three-dimensional model of a given scene from multiple viewpoints. This process typically involves acquiring real-world information through visual sensors and subsequently applying information processing techniques or projection models to extract the three-dimensional structure of objects. 3D reconstruction is of significant research interest within the field of computer vision and constitutes a fundamental focus in machine vision and digital image processing.

Construction methods of 3D maps

In mobile robot manipulation, robust 3D map construction is essential for accurate localization, navigation, and task execution. Existing methods, including OctoMap, NeRF, and 3D Gaussian Splatting (3D GS), exhibit distinct characteristics and trade-offs in terms of representation, computational requirements, and applicability.

OctoMap¹² is an octree-based probabilistic 3D mapping framework capable of representing occupied, free, and unknown space. Its low computational cost and efficient point cloud processing make it suitable for large-scale environments. However, OctoMap suffers from limited detail resolution and reduced accuracy in complex scenes, and it encounters voxel-mesh issues during ray-casting. Enhancements such as OctoMap-RT employ GPU–CPU hybrid strategies to improve voxel resolution and build speed.

Neural Radiance Fields (NeRF)^13–23 leverage implicit neural representations to reconstruct high-fidelity photorealistic scenes. Variants such as MP-NeRF¹³, SNeRG¹⁴, Kilo-NeRF¹⁵, AdaNeRF¹⁶, SinNeRF¹⁷, CCNeRF¹⁸, Super-NeRF¹⁹, and Ref-NeRF²⁰ address challenges in scene complexity, dynamic content, single-image reconstruction, surface smoothness, and cross-view consistency. Dynamic scene reconstruction is further advanced by KFD-NeRF²¹ and optical flow–guided methods²². Despite these improvements, NeRF-based approaches are limited by high computational complexity and long training times, constraining their applicability in real-time mobile manipulation.

3D Gaussian Splatting (3D GS)^24–31 provides a practical alternative for real-time applications. By representing scenes with compact 3D Gaussian primitives and performing block-parallel rasterization, 3D GS achieves fast training and rendering while maintaining high scene fidelity. Extensions such as MPGS²⁵, O-GS²⁷, GaussianPro²⁸, Mini-Splatting²⁹, FreGS³⁰, and GES³¹ further enhance rendering quality, scene detail, and computational efficiency. 3D GS has also been integrated into SLAM systems, including HumanGaussian³², GAPSLAM³³, and CG-SLAM³⁴, enabling robust real-time mapping and manipulation in dynamic indoor environments.

To improve semantic awareness, Semantic SLAM approaches such as NEDS-SLAM³⁵, SGS-SLAM³⁶, and DFS-SLAM³⁷ integrate semantic features into 3D GS representations. These systems support semantic mapping, object-level localization, and manipulation, mitigating cumulative errors and enhancing robustness in dynamic scenes.

Tables 2 and 3 provide a systematic comparison of these approaches. Table 2 evaluates representative SLAM pipelines, including ORB-SLAM3³⁸, CG-SLAM³⁴, and NEDS-SLAM³⁵, considering front-end tracking, back-end optimization, loop closure, map outputs, hardware requirements, and key functionalities. Table 3 analyzes standalone scene representation methods—OctoMap, NeRF, and 3D GS—comparing reconstruction capabilities, scene fidelity, computational complexity, advantages, limitations, and roles within SLAM pipelines.

Table 2.

A representative complete SLAM systems for 3D mapping.

Aspect	ORB-SLAM338	CG-SLAM34	NEDS-SLAM35
Core technology	Feature-based visual-inertial odometry + graph optimization	Uncertainty-aware 3D Gaussian field	3D GS + semantic feature compression
Map output	Sparse 3D points	Dense 3D Gaussian field	Semantic 3D Gaussian primitives
Hardware	CPU	GPU	GPU
Key feature	Multi-map support, robust tracking	High consistency, geometric stability	Semantic mapping, object estimation

Table 3.

Comparison of representative scene representation and rendering methods for 3D mapping: OctoMap, NeRF, and 3D GS.

Comparison project	Octomap39	NeRF40	3D GS24
Method of expression	Octrees	MLP	3D Gaussian element
Scene reconstruction capability	Can efficiently process point clouds41, suitable for large-scale environment map construction, but with slightly weaker detail representation	Can reconstruct highly realistic scenes, but processing dynamic or large-scale scenes is challenging	Can flexibly simulate objects, with advantages in efficiency and adaptability
Computational complexity	It can reduce memory consumption42 and has relatively low computing costs, making it suitable for scenarios with high real-time requirements	High computational complexity, time-consuming training and rendering	The complexity lies between the two, requiring a certain amount of computing resources
Advantages	Balances real-time performance and compression efficiency, suitable for large-scale environment mapping	Neural implicit fields enable highly realistic view synthesis	3D Gaussian primitives achieve a flexible balance between quality and efficiency
Disadvantages	Weak ability to depict scene details	Training and rendering are computationally expensive and difficult to handle in dynamic, large-scale scenes	Still relies on considerable graphics memory and computing resources
Role in SLAM pipeline	Mapping backend	Scene representation and rendering	Scene representation and rendering

OctoMap, NeRF, and 3D GS are scene representation or mapping backend methods, not complete SLAM systems. Complete SLAM systems (e.g., ORB-SLAM3⁴³, CG-SLAM³⁴) integrate front-end tracking, back-end optimization, and loop closure detection.

In summary, OctoMap is suitable for large-scale, low-detail mapping; NeRF excels in high-fidelity photorealistic reconstruction but is limited in dynamic environments; 3D GS offers a balanced solution, combining efficient rendering, high-quality reconstruction, and real-time applicability, particularly when enhanced with semantic integration. These characteristics make 3D GS-based SLAM frameworks highly promising for mobile robot manipulation in dynamic indoor environments.

In addition to the previously discussed methods, several specialized 3D reconstruction techniques have been developed to address complex or application-specific scenarios. For example, Kim et al.⁴⁴ proposed a real-time pipeline reconstruction method using RGB-D cameras, employing superpixel decomposition, sequential grouping of pipeline elements, and parametric equation modeling to enhance 3D map expressiveness for linear infrastructures. For 3D object recognition in voxel data, Karambakhsh et al.⁴⁵ introduced a hybrid framework combining three compact convolutional neural networks (CNNs), SparseNet, and a feature representation technique, achieving a balance between efficiency and accuracy.

Transformer-based approaches have also been applied to 3D reconstruction⁴⁶. Conventional Transformer methods typically establish 2D-to-3D relationships or perform feature extraction, but often fail to capture deep, multi-scale voxel representations. To overcome this limitation, the Transformer with Multi-Scale Dense Network (TMSDNet)⁴⁷ was proposed, providing enhanced multi-scale feature representation and improving the reliability of robot mobile manipulation in complex environments.

As summarized in Table 4, representative vSLAM methods—including both 3D and semantic mapping approaches—are compared in terms of core technology, applicable scenarios, advantages, and limitations, offering a concise and practical reference for researchers.

Table 4.

Comparison of representative visual SLAM (vSLAM) methods, including both complete SLAM systems and typical scene representation approaches.

Method	Map type	Core technology	Suitable scenarios	Advantages	Limitations
Octomap39	3D map	Octree-based probabilistic mapping	Large-scale environment reconstruction, real-time navigation	Low memory consumption, efficient updates, multi-resolution support	Limited detail representation, sensor-dependent
NeRF40	3D map	Implicit neural representation with MLP	High-quality view synthesis, photorealistic reconstruction	Realistic rendering, handles complex geometry	Slow training and rendering, poor generalization to new scenes
3D GS24	3D map	3D Gaussian primitives with rasterization	Real-time rendering, dynamic scene reconstruction	Fast rendering, high-quality results, editable scenes	High memory usage, limited geometric accuracy
ORB-SLAM343	3D map (complete SLAM system)	Feature-based visual-inertial odometry	General indoor/outdoor navigation, multi-sensor fusion	High accuracy, robust tracking, multi-map support	Computationally intensive, feature-dependent in textureless areas
DS-SLAM67	Semantic map	Semantic segmentation + motion consistency check	Dynamic indoor environments, human–robot interaction	Strong semantic understanding, robust to moving objects	Requires labeled data, limited generalization

Complete SLAM systems are indicated accordingly.

Discussion and analysis of 3D maps

In summary, the construction of 3D maps plays a critical role in supporting autonomous navigation and environmental perception for mobile robots across diverse scenarios. OctoMap, based on an octree structure with probabilistic node representations, offers low storage requirements, efficient data updates, and multi-resolution mapping capabilities. However, in complex dynamic environments, its performance is highly dependent on sensor quality and computational resources. Neural Radiance Fields (NeRF) enable reconstruction of complex scenes without precise geometric models and generate photorealistic 3D renderings, facilitating high-fidelity mapping. Yet, NeRF suffers from slow rendering, high training data requirements, and limited generalization, often necessitating retraining for novel scenes. 3D Gaussian Splatting (3D GS) provides fast rendering, high-quality scene representation, and easy scene manipulation, making it particularly suitable for online SLAM and real-time 3D map construction. Nonetheless, its geometric accuracy and measurability remain limited, restricting applicability in tasks that require precise spatial reasoning.

Despite its advanced rendering capabilities, direct application of 3D GS to robot manipulation must account for physical constraints. For collision avoidance and motion planning, the map representation must support rapid occupancy queries, distance computation, and differentiable collision checking. Current 3D GS methods excel in novel view synthesis but do not inherently provide signed distance fields or occupancy probabilities, which are essential for safe motion planning.

Recent efforts have begun addressing these limitations. SPLANNING⁴⁸ introduces a risk-aware trajectory optimizer operating directly on Gaussian Splatting models. By deriving a rigorous upper bound on collision probabilities and employing a normalized reformulation for computational efficiency, SPLANNING enables collision-free trajectory optimization in cluttered environments. Experimental validation with a real-world robot manipulator demonstrates its effectiveness, highlighting a promising pathway for integrating 3D GS into task-oriented SLAM systems for physical execution. Nevertheless, challenges remain: computational demands are substantial, and robustness under environmental disturbances and varying lighting conditions requires further improvement.

Task suitability of 3D maps

3D maps provide metric accuracy and geometric occupancy information, making them inherently suitable for robot navigation and obstacle avoidance. Their precise spatial representation enables accurate localization and collision-free path planning in complex environments. As illustrated in Fig. 6, a mobile robot navigates toward the target while avoiding moving cylindrical obstacles in an unknown dynamic environment⁴⁹. For such tasks, OctoMap-based representations achieve a favorable balance between computational efficiency and geometric fidelity. In large-scale environment reconstruction, octree-based methods ensure efficient storage, while 3D Gaussian Splatting (3D GS) enables fast rendering for real-time applications.

Fig. 6.

A representative dynamic obstacle avoidance scenario for mobile robot navigation, showing a robot navigating toward the target while avoiding four moving cylindrical obstacles⁴⁹.

However, 3D maps alone are insufficient for object manipulation tasks, as they lack the object-level semantic information required for grasp pose estimation. This limitation motivates the adoption of semantic and object-level maps for manipulation scenarios that involve direct physical interaction with objects. Consequently, a task-oriented trade-off arises: for navigation on resource-constrained platforms, pure 3D maps remain the most practical option, whereas for tasks involving object interaction, 3D maps need to be augmented with semantic or object-level information, as discussed in “Semantic maps” and “Object-level maps” sections.

Semantic maps

3D maps provide a “spatial skeleton” of dynamic indoor environments through precise geometric modeling. However, they are limited to representing object positions and environmental changes and do not inherently differentiate between object types or explain the causes of scene variations. Consequently, 3D maps alone cannot support robots in understanding object categories or the logic underlying dynamic changes during navigation or manipulation.

To overcome this limitation, semantic maps have been developed. Building upon 3D maps, semantic maps assign semantic labels to objects, regions, and dynamic events within the three-dimensional space. These maps not only preserve the geometric information provided by traditional maps—such as object locations, shapes, and spatial relationships—but also incorporate rich semantic attributes, including object categories, scene functions, and other contextual properties.

By integrating geometric and semantic information, semantic maps provide robots with a more sophisticated understanding of their environment, enabling more informed decision-making, planning, and improved efficiency and accuracy in movement and task execution. As illustrated in Fig. 7, a 3D semantic map construction algorithm can generate a globally consistent 3D semantic map⁵⁰ in a laboratory environment, accurately labeling the semantics of various objects and demonstrating the effectiveness of semantic augmentation in practical robotic scenarios.

Fig. 7.

Results of reconstructed 3D semantic maps⁵⁰.

Semantic mapping technology support

The construction of semantic maps relies fundamentally on semantic information, which refers to concepts that can be intuitively perceived and interpreted by humans. In the context of images, semantic information encompasses object contours, categories, and significance within a scene. Semantic vSLAM leverages this information to enhance both localization accuracy and robustness while simultaneously improving environmental perception through object-level reconstruction.

In a semantic map, each map point is associated with a specific target object, and each keyframe is indirectly linked to one or more target objects via these map points. This association establishes a structured correspondence between the image data and the objects in the environment, enabling robots to interpret scenes at a higher semantic level and to perform tasks that require object-specific understanding, such as targeted manipulation or context-aware navigation.

Construction methods of semantic maps

Sensor fusion methods integrate multi-source information through loose or tight coupling schemes, enhancing geometric accuracy while simultaneously extracting semantic features from images. This dual capability enables robots to perceive their environment more accurately during mobile manipulation. Representative examples include Nvblox⁵¹, which leverages NVIDIA’s CUDA parallel computing architecture to process RGB-D data, integrating depth and color information to build real-time 3D voxel maps while performing semantic segmentation and object recognition. VDBblox⁵² fuses RGB-D and LiDAR data to construct Euclidean Signed Distance Fields (ESDF) incrementally from Truncated Signed Distance Fields (TSDF), improving mesh reconstruction quality. Similarly, Voxfield⁵³ and VINS-based methods⁵⁴ integrate multi-sensor data to generate high-quality maps and robust monocular visual-inertial navigation, demonstrating strong adaptability to dynamic indoor environments. Despite these advantages, sensor fusion approaches face challenges when processing complex scenes or large-scale datasets, and their semantic understanding remains limited by the inherent capabilities of individual sensors.

Deep learning-based methods address these limitations by automatically extracting complex semantic features from large labeled datasets. Techniques combining Mask R-CNN with SLAM, such as the indoor 3D semantic vSLAM proposed by Tao et al.⁵⁵, perform real-time object detection and instance segmentation, reducing the impact of dynamic objects. IEMask R-CNN⁵⁶ further improves instance classification accuracy and mask quality by preserving channel information and refining features, achieving substantial benchmark gains on MS COCO2017, Cityscapes, and LVIS1.0 datasets. Other variants, such as edge-enhanced Mask R-CNN⁵⁷, probabilistic Faster R-CNN⁵⁸, and YOLO-based approaches (e.g., ERF-YOLO⁵⁹, YOLO*C⁶⁰, EdgeTrim-YOLO⁶¹), demonstrate improvements in segmentation completeness, inference speed, and contextual understanding. Deep learning-based SLAM strategies such as RGB-PD⁶² and MD-SLAM⁶³ leverage predicted depth maps and multi-task networks to provide scale-consistent depth estimation and dynamic object detection, enhancing environmental perception in complex indoor scenarios.

Probabilistic graphical model-based methods are particularly useful for handling sensor noise and incomplete data by performing probabilistic reasoning to infer more accurate semantic labels. Examples include CP-SLAM⁶⁴, which integrates historical associations with feature points and semantic information to transform target detection into a probabilistic graph, improving indoor localization accuracy by approximately 20% on the TUM dataset. RDP-SLAM⁶⁵ mitigates dynamic object interference by removing associated feature points, reducing absolute trajectory error (APE) in ORB-SLAM3 by 80% in highly dynamic environments. GGC-SLAM⁶⁶ combines predicted static probabilities with conditional random fields to refine dynamic-static distinctions, significantly improving trajectory estimation and mobile manipulation accuracy.

In summary, sensor fusion compensates for the limitations of individual sensors through multi-modal integration, deep learning provides rich semantic understanding through learned feature extraction, and probabilistic graphical models address uncertainty in complex and dynamic scenes. Their complementary strengths make these methods suitable for different semantic map construction scenarios. To clarify their respective characteristics across key dimensions—accuracy, robustness, real-time performance, and semantic richness—we summarize a comparative analysis in Table 5.

Table 5.

Comparison of semantic map construction methods—including sensor fusion, deep learning, and probabilistic graphical models—across representation approaches, data requirements, scene reconstruction capabilities, and computational complexity.

Comparison project	Based on sensor fusion51–54	Based on deep learning55–62	Based on probabilistic graphical models64–66
Method of expression	Fusion of multi-sensor data	Representing the environment using point clouds and assigning semantics	Deep models
Data requirements	Real-time data from multiple sensors	A large amount of labeled images or point cloud data	Prior knowledge or a small number of samples combined with sensor data
Scene reconstruction capability	Skilled in geometric reconstruction, but semantic understanding is limited by sensors	Strong semantic understanding, but limited in complex scenarios	Good at handling uncertainties
Computational complexity	Multi-source data processing involves a large amount of computation	Training consumes resources and has certain requirements for reasoning ability	Precise reasoning is complex and requires approximate algorithms

Recent advances have significantly enhanced semantic mapping in dynamic environments. Dense semantic methods enable per-pixel labeling for fine-grained scene understanding, while multi-feature fusion combining point and line features provides additional geometric constraints in texture-poor regions. A representative example is Dynamic Point and Line Features SLAM (DPL-SLAM)⁶⁸, which integrates target detection with optical flow and epipolar constraints to identify dynamic features. By combining dense semantic information with geometric features, DPL-SLAM illustrates how semantic understanding can guide robust feature selection in dynamic scenes.

Beyond SLAM, feature selection techniques further leverage semantic guidance. Mutual Information-guided Angle Reconstruction and Semantic Alignment (MIARS)⁶⁹ improves feature discriminability through semantic alignment. Multi-label Feature Selection with High-level Semantic Relationships (HSR-MLFS)⁷⁰ uses inter-label relationships to guide feature selection, while Hierarchical Feature Selection Driven by Inter-category Relevance and Semantic Independence (HFSIS)⁷¹ preserves semantic independence by weakening highly dependent feature relationships. Collectively, these methods demonstrate the utility of semantics in guiding geometric feature processing.

In addition to semantic-guided selection, two complementary mechanisms have emerged to handle dynamics more robustly: motion-consistency reasoning and probabilistic dynamic filtering. Motion-consistency reasoning exploits geometric constraints across frames to verify whether semantically dynamic objects are truly moving. Unlike early epipolar-based approaches, which fail to detect motion along the viewing direction, Trifocal SLAM⁷² introduces a three-frame coplanarity constraint, enabling reliable separation of static and dynamic points even for objects moving directly toward the camera. Integrated with YOLOv8-seg semantic segmentation, it achieves up to 98.38% improvement in RMSE of absolute trajectory error over ORB-SLAM2 on public datasets (TUM RGB-D, BONN).

Probabilistic dynamic filtering maintains uncertainty estimates for map elements, supporting informed decisions about dynamic content. Bayesian Dynamic Gaussian Splatting SLAM (BDGS-SLAM)⁷³ constructs a Bayesian dynamic prior for Gaussians and employs multi-view probabilistic updates with exponential decay, restoring static elements erroneously removed.

Together, semantic-guided feature selection, motion-consistency reasoning, and probabilistic dynamic filtering represent a shift from reactive dynamic object removal to proactive reasoning about scene dynamics, substantially enhancing robustness in real-world mobile manipulation and navigation tasks.

Advantages and disadvantages of semantic maps

Semantic maps offer several key advantages. First, compared with purely geometric image features, semantic information exhibits greater temporal and spatial consistency and aligns more closely with human perception. Second, integrating semantic information enhances the positioning accuracy and robustness of vSLAM while enriching environmental perception. Third, semantic maps are easily updated and maintained: when objects or environmental elements change, local updates based on semantic labels can be performed, reducing computational load and time.

However, semantic maps also have limitations. One challenge lies in the optimization framework for objects. Current approaches primarily adopt either tightly coupled or loosely coupled schemes, each suited to specific scenarios. Notably, a tightly coupled scheme does not always improve localization accuracy and can sometimes degrade it. For instance, DS-SLAM⁶⁷, proposed in 2018, combines semantic segmentation with motion-consistency checking. Experiments on the TUM RGB-D dataset and real-world environments showed that the loosely coupled DS-SLAM achieves an order-of-magnitude improvement in absolute trajectory accuracy compared to tightly coupled ORB-SLAM2, a feature-based monocular SLAM system. Another limitation is the reliance on high-precision mapping equipment, such as LiDAR, for constructing prior maps, which increases cost and limits scalability. Consequently, further research is needed to develop semantic mapping solutions that are more adaptable to complex mobile manipulation tasks.

Task suitability of semantic maps

Semantic maps extend geometric representations by incorporating categorical labels and scene functions, enabling robots to understand not only object locations but also their identities. This capability is particularly valuable for human–robot collaboration and task-oriented navigation.

As illustrated in Fig. 8, in a human–robot collaborative indoor scenario, robots equipped with semantic maps can safely navigate around humans by recognizing labels such as “person,” “workspace,” and “social zone”⁷⁴. For such applications, Zheng et al.⁷⁵ proposes a semantic map construction method that integrates semantic and geometric information through three modules: semantic information generation, geometric information generation, and semantic-geometric fusion. The semantic module removes dynamic object features, and together, these modules enhance localization accuracy in dynamic environments. For task-oriented navigation, semantic labels guide robots to functional areas—such as kitchens, doorways, or storage zones—enabling goal-directed actions. For example, a robot tasked with fetching a cup can navigate directly to the kitchen rather than searching the entire environment.

Fig. 8.

Human–robot collaborative interaction based on semantic map in indoor environments⁷⁴.

Despite these advantages, semantic maps are limited for fine-grained manipulation tasks that require instance-level precision. While they identify object categories (e.g., “cup”), they do not distinguish individual instances or provide the precise 6-DoF poses required for grasp planning. Real-time deployment on low-power devices is also challenging due to the computational demands of semantic segmentation networks.

Thus, a clear guideline emerges: for tasks requiring high-level scene understanding—such as human–robot collaboration, service robotics, or assistive applications—semantic maps are essential. For resource-constrained platforms, lightweight semantic networks^76–78 provide a practical trade-off between semantic richness and computational efficiency. However, for precise object manipulation, semantic maps should be augmented with object-level information, as discussed in “Object-level maps” section.

Object-level maps

Object-level maps represent a refinement of semantic maps, shifting from category-level recognition to instance-level perception. While semantic maps provide information on object categories and regional functions, object-level maps assign unique identifiers, attribute databases, and real-time status information to individual objects. This enables robots to understand detailed scene semantics, respond effectively to frequent changes, and meet complex demands in dynamic indoor environments, thereby improving the efficiency and accuracy of mobile manipulation.

As high-granularity representations, object-level maps go beyond spatial regions by specifying detailed object properties, including location, category, and attributes such as size, color, and orientation. For instance, an indoor map may capture the precise placement and characteristics of items on a table. Constructing these maps relies on advanced sensors and target recognition algorithms, which allow mobile robots to perceive their surroundings accurately and execute task-oriented manipulations more effectively.

Object-level map technology support

Object-level maps encode detailed information on object attributes and inter-object relationships, enabling robots to accurately identify and manipulate specific objects, thereby enhancing the efficiency and precision of tasks such as object retrieval. These maps leverage data associations across multiple frames to integrate detection results and initialize object representations. In the back-end optimization process, both camera poses and object poses are refined, while object model parameters are adjusted. Through this procedure, the system constructs a comprehensive object-level map, providing semantic-rich perception of the environment for task-oriented mobile manipulation.

Methods for constructing object-level maps

Object-level SLAM, which integrates object detection and instance segmentation, has emerged as a prominent approach in object-level mapping research. This method leverages multi-view geometric constraints and detection bounding boxes to reconstruct object models in mobile manipulation scenarios. Several systems and improvements have been proposed. MaskFusion², for instance, is a real-time, object-aware, semantic RGB-D SLAM system that surpasses traditional static, geometry-only mapping approaches. Unlike earlier recognition-based SLAM systems, MaskFusion does not require prior object models and utilizes instance-level semantic segmentation to incorporate semantic labels into object maps. Its convolutional mask module operates asynchronously at 5 Hz, imposing significant computational demands and necessitating a dedicated GPU for real-time performance (Fig. 9).

Fig. 9.

The renderings generated in a dynamic indoor environment. It reflects MaskFusion’s recognition, tracking, and mapping capabilities².

ASF-YOLO⁷⁹ employs attentional scale sequence fusion within a YOLO framework, integrating the Scale Sequence Feature Fusion (SSFF) and Triple Feature Encoder (TFE) modules to enhance multi-scale feature extraction. On the 2018 Data Science Bowl dataset, it achieves a box mAP of 0.91, mask mAP of 0.887, and an inference speed of 47.3 FPS. CoT blocks⁴⁶ further exploit static and dynamic contextual information, enhancing feature representation. OA-SLAM⁸⁰ demonstrates improved SLAM repositioning in unknown environments through advanced object landmarks, while Song et al.⁸¹ utilize object-level landmarks and prior size constraints to reduce scale drift and improve monocular localization accuracy and robustness.

Another common approach combines deep learning with point cloud processing. This strategy uses advanced networks for semantic and instance segmentation on 3D point clouds, often supplemented with IMU data to enhance pose estimation and adapt to dynamic environments. For example, PointSeg⁸² performs instance segmentation on images via YOLACT +  + ⁸³ and associates the results with point clouds, achieving instance-level segmentation with ~ 75 ms for images and ~ 178 ms per point cloud instance on standard hardware. LO-Net⁸⁴ applies a layer-wise feature optimization network for accurate point cloud classification. CrossFusion Net⁸⁵ fuses lidar and RGB features hierarchically to improve environmental perception. Other methods, such as automated BIM generation⁸⁶, HPCT⁸⁷, and PointFilterNet (PFN)⁸⁸, enhance semantic segmentation, feature extraction, and noise reduction. Chen et al.⁸⁹ address adversarial vulnerabilities in point cloud processing, improving safety and reliability in robotic manipulation.

For resource-constrained deployment, lightweight networks such as MobileNet variants have been explored. Thin MobileNet⁹⁰ reduces model size to 9.9 MB using separable convolutions, Slim MobileNet⁹¹ achieves 7.3 MB while improving accuracy, and MobileNet-SA⁹² leverages self-attention mechanisms to reach 93.5% accuracy on the SketchyCOCO dataset. NodeSLAM⁹³ introduces multi-class object descriptors and a differentiable probabilistic rendering engine for robust 3D reconstruction. DynaQuadric⁹⁴ proposes tightly coupled optimization for quadric surface representation combined with multi-object tracking (MOT), enabling object perception in dynamic environments. KN-SLAM⁹⁵ integrates keypoint localization with neural implicit representations, achieving high-quality reconstruction while maintaining precise localization through global and local feature fusion.

To systematically compare these approaches, a table summarizing fusion-based object detection and segmentation methods versus deep learning-based point cloud processing methods is provided in Table 6.

Table 6.

Comparison of methods integrating fusion-based object detection and instance segmentation with deep learning-based point cloud processing, including representation approach, data requirements, scene understanding capabilities, and computational complexity.

Comparison project	Integrating object detection and instance segmentation2,46,79–81	Deep learning combined with point clouds82–89
Method of expression	First detect and locate, then segment, and represent the scene with target instances	Use models to process point clouds and learn feature descriptions of scenes
Data requirements	Annotated image data	Point cloud with annotations, or combined with RGB information
Scene understanding	Strong target recognition, weak geometric understanding	Good geometric perception, weak instance segmentation
Computational complexity	Processing high-resolution images requires a lot of computing power	Due to irregular point clouds, calculations are complex

Object-level mapping methods help reduce cumulative errors and provide more accurate pose estimates and environmental models for mobile manipulation. In this context, ObjectFolder⁹⁶ is a dataset comprising 100 virtual objects, encoding visual, auditory, and tactile sensory data for each object. It employs a unified, object-centric implicit representation to capture visual textures, acoustic simulations, and tactile readings, facilitating easy usage and sharing. However, due to its limited scale and the restricted quality of multisensory data, ObjectFolder faces challenges in generalizing to real-world scenarios. The introduction of ObjectFolder 2.0⁹⁷ addresses these limitations by providing a large-scale, multi-sensory dataset of home objects represented via implicit neural networks, significantly enhancing the capabilities and applicability compared to ObjectFolder 1.0.

Advantages and disadvantages of object-level maps

In dynamic indoor environments, object-level maps offer several advantages for mobile manipulation. First, incorporating semantic attributes from object detection into vSLAM improves localization accuracy by distinguishing static from dynamic objects, effectively mitigating interference from moving elements. For instance, OA-SLAM⁸⁰ integrates feature descriptors and object reprojection observations, leveraging relative object position constraints to achieve robust localization under dramatic perspective changes. Similarly, object-level and deep network descriptors enhance localization robustness in textureless or challenging scenarios. To improve environmental perception, some studies construct quadratic surface maps of objects⁹⁴, simultaneously estimating size, orientation, and position. Others incorporate dynamic object information into SLAM, as exemplified by OTE-SLAM⁹⁸, which tracks both camera and object motion, providing a more reliable basis for decision-making in mobile manipulation.

Despite these advances, object-level maps face significant practical challenges. First, their computational demands are high: introducing semantic attributes, constructing object databases, and performing joint optimization require high-performance hardware, limiting deployment on resource-constrained platforms. Second, object detection algorithms remain vulnerable to complex scenarios; abrupt lighting changes or severe occlusions can lead to recognition errors, adversely affecting localization. Third, map construction is costly, as data collection and processing are time-intensive, posing obstacles for large-scale applications. Existing methods often lack versatility, being optimized for specific scenarios and requiring significant adaptation for new environments.

To address these limitations, cross-scenario adaptation strategies have been explored. Transfer learning can leverage models trained in one environment to other scenarios, reducing repetitive training. For example, Henein et al.⁹⁹ incorporated meta-structural information related to geometric primitives into SLAM, enabling accurate online estimation of environmental structure and robot pose. Zhuang et al.¹⁰⁰ evaluated over 20 transfer learning models, demonstrating the importance of model selection for practical deployment. Local instance-based transfer learning for reinforcement learning¹⁰¹ uses Q-values of local trajectories to evaluate similarity, significantly improving transfer efficiency and enabling rapid adaptation in dynamic indoor environments. Domain-adaptive strategies bridge distribution gaps between datasets. Huang et al.¹⁰² proposed a semi-supervised domain adaptation (SSDA) method—bidirectional sample-class alignment (BSCA)—for remote sensing cross-domain scene classification, achieving compact feature representation and low-entropy classification boundaries, thereby providing reliable environmental information for mobile manipulation.

Task suitability of object-level maps

Object-level maps assign unique identities, dynamic attributes, and precise geometric models to individual objects. This fine-grained representation makes them particularly suited for robotic tasks that require precise interaction with specific objects, such as grasping and persistent target tracking in dynamic environments. As illustrated in Fig. 10, object-level maps provide not only spatial locations but also accurate poses, dimensions, and contour profiles of objects¹⁰³. Such high-precision perceptual information is essential for grasp planning and force-closure computation. Unlike conventional 3D metric maps, which represent the environment as undifferentiated geometric primitives, object-level maps enable robots to discriminate individual entities, maintain long-term object tracking, and devise manipulation strategies tailored to the physical characteristics of each object. In dynamic scenes, they enhance SLAM robustness by decoupling static background elements from moving objects. For example, OA-SLAM⁸⁰ achieves reliable localization under significant perspective variations by fusing object feature descriptors with reprojection constraints.

Fig. 10.

Robotic grasping based on object-level map information¹⁰³.

Despite their advantages for fine manipulation, object-level maps face limitations in large-scale navigation or environments with numerous unseen objects. Their construction and maintenance impose substantial computational overhead, including high GPU usage and memory consumption, which reduces efficiency in large-area navigation tasks where lightweight geometric maps suffice. Moreover, most object-level mapping frameworks rely on pre-trained detection models that may fail when encountering novel object categories, limiting generalization.

Consequently, a task-oriented selection principle is evident. Object-level maps are recommended for manipulation tasks requiring precise interaction with specific targets, such as grasping, assembly, or object fetching. Deployment on resource-limited platforms necessitates hardware acceleration or model compression, while transfer learning strategies^99–102 can improve cross-scenario generalization. Conversely, standard 3D metric maps remain more cost-effective for large-scale navigation or tasks that do not require object-level discrimination.

Consider a robot tasked with retrieving a designated cup from a cluttered table. Using a standard 3D map, the robot can navigate to the table but cannot identify the target cup among surrounding objects, resulting in grasp failure. In contrast, an object-level map provides the precise pose, size, and shape of the cup, enabling correct object recognition, optimized grasp planning, and task completion. This example underscores the indispensability of object-level maps for interactive manipulation, while highlighting the need to balance semantic richness with computational resources based on task requirements.

Hybrid maps

The evolution from 3D maps to semantic maps and finally to object-level maps represents a progressive refinement from geometric representation to semantic understanding to instance-level perception. To fully exploit the strengths of each map type, hybrid maps have been proposed, enabling multi-level integration of spatial, semantic, and object-specific information. A hybrid map combines diverse representation methods or data modalities to provide a more comprehensive and accurate depiction of the environment. By integrating geometric elements (e.g., points, lines, voxels), semantic labels and topological relationships, and multi-modal sensor data (e.g., LiDAR, RGB-D), hybrid maps overcome the limitations of individual map types. This multi-layered representation enhances environmental understanding and provides robust support for mobile manipulation in dynamic indoor scenarios.

Classification of hybrid maps

Hybrid maps integrate multiple map representation methods or data modalities to achieve a more comprehensive and accurate depiction of environmental information. As illustrated in Fig. 11, in dynamic indoor environments, hybrid maps can be categorized into hierarchical maps and multimodal maps. Hierarchical maps combine local metric details with global topological structures. At the local level, high-precision laser-scanned point clouds capture fine-grained spatial information within confined areas, supporting accurate positioning and motion planning. The global topology abstracts indoor spaces as nodes and edges, representing inter-area connectivity and facilitating efficient global path planning. This hierarchical integration allows robots to execute detailed tasks while maintaining an understanding of the overall layout.

Fig. 11.

Classification map of the hybrid map.

Multimodal maps, on the other hand, fuse visual data with LiDAR information¹⁰⁴. Vision provides rich texture, color, and semantic cues for object recognition and scene interpretation, whereas LiDAR delivers precise 3D geometry and distance measurements. By combining these modalities, multimodal maps simultaneously capture the environment’s geometric structure and semantic content, furnishing a robust foundation for mobile manipulation and enabling robots to perceive and adapt effectively to dynamic indoor environments.

Application and improvement of hybrid maps

Hybrid maps are typically constructed using point and planar features. They first extract feature points and planes, associate data through decision trees, and apply smoothing mapping tools to construct factor graphs for joint optimization of robot poses and landmarks, enabling real-time updates. Such maps form the basis of indoor navigation systems, providing critical support for mobile manipulation tasks.

Early hybrid mapping efforts include the work of Tomatis et al.¹⁰⁵, which integrated metric and topological paradigms using a 360° laser scanner, achieving multi-modal topological localization without requiring global metric consistency. Subsequent developments employed Rao-Blackwellized particle filters for real-time data association, mapping, and spatial reasoning with hybrid map representations¹⁰⁶. Konolige et al.¹⁰⁷ combined topological maps with local occupancy grids for large-scale navigation, while Choi et al.¹⁰⁸ extended RBPF-based hybrid SLAM to the Velodyne 3D HDL-64 scanner, providing rich spatial information. OGI-SLAM2¹⁰⁹ and Kimera2^110,111 further improved localization accuracy through optimized keyframe selection, multiple input modes, and enhanced pose map backends.

Recent advances in hierarchical maps focus on increasing accuracy and efficiency. Kong et al.¹¹² proposed a points-of-interest-driven method to extract local topological features, while GTCAlign¹¹² encodes node embeddings for graph alignment guided by global topology. H-SLAM¹¹³ uses descriptor sharing to generate a single inverse depth representation, and UL-SLAM¹¹⁴ employs multi-scale line matching to reduce motion errors.

Multimodal hybrid maps have also advanced significantly. Dynam-SLAM¹¹⁵ integrates dynamic and static features with IMU measurements for nonlinear optimization. MSSKF¹¹⁶ and FMF¹¹⁷ improve multi-modal fusion for object detection and small-target perception. Filter Fusion¹¹⁸ couples camera and LiDAR features, while TextSLAM¹¹⁹ incorporates text objects as semantic priors for mixed-reality navigation. ESLAM¹²⁰ uses signed distance field representations with NeRF integration to enhance dense SLAM efficiency, and AVLMaps¹²¹ leverage audio-visual-language cues for multi-level indexing. HERO-SLAM¹²² combines neural implicit fields with feature-metric optimization, and HVL-SLAM¹²³ integrates visual-LiDAR data for robust tracking. Ji et al.¹²⁴ introduced object-level SLAM using spatial layout and semantic consistency for loop closure and manipulation accuracy. Local control-based multisensor fusion¹²⁵ mitigates cumulative trajectory errors from asynchronous sensor measurements.

A summary comparison of hierarchical and multimodal maps is provided in Table 7.

Table 7.

Comparison of hierarchical and multimodal hybrid maps, highlighting their definitions, data sources, and key advantages.

Comparison project	Hierarchical maps105–114,126	Multimodal maps115–125
Definition	Mixing local metrics and global topology	Visual and LiDAR hybrid
Data	Laser point cloud, node edge structure	Visual and LiDAR data
Advantages	Local precision, global grasp of layout	Balancing texture and geometric information
Disadvantages	Closed-loop detection is susceptible to changes in perspective and repetitive scenes, which can cause cumulative errors	Multi-sensor synchronization and data correlation are complex and computationally intensive

A critical aspect of hybrid mapping in dynamic environments is the joint utilization of semantic and structural features to mitigate dynamic interference. Rather than treating semantics as an independent layer, recent approaches^127–129 integrate semantic understanding directly into geometric mapping pipelines. This integration enables the system to more reliably distinguish between static and dynamic elements. For instance, semantic labels can identify potentially moving objects (e.g., humans, vehicles), while structural features—such as points, lines, and planes—provide geometric consistency checks. When a semantic label indicates an object is dynamic but geometric features remain consistent, the system can probabilistically decide whether to include or exclude that element from the map.

This synergy is particularly effective in multimodal mapping, where visual data is combined with other sensor modalities such as LiDAR or IMU. Semantic segmentation can be applied across modalities, enabling cross-modal validation of object dynamics. Structural features extracted from different sensors provide complementary information: visual data contributes rich texture and color, while LiDAR supplies precise depth and geometric structure. By fusing these modalities under semantic guidance, hybrid maps achieve robust performance even under challenging conditions, including lighting variations, occlusions, or rapid motion. A notable example is SLAM2¹³⁰, a semantic RGB-D SLAM system that integrates point, line, and planar features with deep learning to detect both known and unknown dynamic objects, demonstrating enhanced robustness in dynamic environments.

Thus, combining semantic understanding with multimodal structural features provides a principled approach to mitigating dynamic interference. Leveraging the strengths of each modality and using semantics to guide feature selection and fusion allows hybrid maps to maintain consistency and accuracy where traditional methods often fail.

Hybrid maps have been extensively applied in robotic localization and mapping. They integrate metric and topological paradigms by connecting local metric maps through global topological structures to form compact environment representations. Using 360° laser scanners and visual features, hybrid maps can handle environmental loops and support precise positioning in unknown spaces. Their advantages are particularly evident in complex dynamic environments. For example, in multi-floor warehouse navigation, a single metric map is insufficient due to the large environment, complex cargo layouts, and frequent movement of personnel and equipment. Hybrid maps overcome these limitations by combining local metric maps, which capture precise dimensions and locations of shelves and aisles, with global topological maps, which encode floor-to-floor connections such as staircases and elevators. This combination enables robots to navigate accurately through narrow aisles, avoid obstacles, and plan optimal paths across floors, thereby supporting efficient transportation and sorting operations.

Advantages and disadvantages of hybrid maps

Hybrid maps, constructed from point and plane features, leverage the complementary advantages of each representation to enhance navigation practicality. They integrate diverse types of information, combining geometric features for intuitive path planning with connectivity features that capture the overall structural relationships of the environment. This multi-level representation also makes hybrid maps well-suited for complex manipulation tasks requiring both geometric precision and semantic understanding.

Despite their strengths, hybrid maps face inherent limitations. The increasing algorithmic complexity can hinder stable operation across large and diverse environments, complicating map construction. Additionally, current hierarchical SLAM approaches lack standardized map model conversion methods, creating challenges in data processing and interoperability. Future research should focus on reducing algorithmic complexity, unifying model transformation protocols, and improving the accuracy and efficiency of data association.

Task suitability of hybrid maps

Hybrid maps integrate multiple representation types within a unified framework, combining geometric, semantic, topological, and multimodal information. This integration makes them particularly suitable for complex manipulation tasks that require both geometric precision and semantic understanding.

As illustrated in Fig. 12, which depicts top-down pick-and-place manipulation using hybrid maps, these maps merge fine-grained object geometry from object-level representations with scene semantics to identify safe manipulation paths and avoid collisions¹³¹. For long-term operation in dynamic environments, hybrid maps support efficient static-dynamic distinction and selective map updating, maintaining global consistency while updating only regions affected by changes. This capability is essential for robots operating over extended periods in environments with moving people or rearranged furniture. In multi-floor navigation scenarios, such as warehouse operations, hybrid maps leverage local metric maps for precise positioning on each floor while preserving global topological consistency for cross-floor path planning.

Fig. 12.

Top-down pick-and-place manipulation using hybrid maps. Object-level geometry is fused with scene semantics to enable collision-free grasping and accurate placement in cluttered environments¹³¹.

Despite these advantages, hybrid maps are less suitable for deployment on highly resource-constrained platforms or latency-sensitive applications. They require multiple sensors (cameras, LiDAR, IMU) and GPU acceleration, making deployment on low-power platforms challenging. Real-time multimodal data synchronization and fusion remain computationally intensive, and current hierarchical SLAM approaches still lack standardized map model conversion methods, complicating data processing.

Accordingly, hybrid maps are recommended for scenarios that demand both high geometric accuracy and rich semantic understanding, such as cluttered pick-and-place tasks or long-term autonomous operation in dynamic indoor environments. For simpler applications, such as basic navigation or coarse grasping, a single map type—either 3D or object-level—may suffice and be more computationally efficient.

In conclusion, 3D maps have been extensively studied and provide the foundation for environmental representation. Recent research has increasingly focused on semantic and object-level maps to enable higher-level scene understanding and task-oriented manipulation. Hybrid maps, integrating multiple representation modalities, represent an emerging direction with significant potential for future exploration.

Table 8 summarizes the performance characteristics and hardware requirements of representative methods for each map type, based on the discussions in “3D maps” to “Hybrid maps” sections. Table 9 highlights the core research focus, key technologies, current challenges, and future directions for each map type, drawn from the technical literature reviewed.

Table 8.

Performance metrics and hardware requirements of representative methods across four map type.

Method	Type	Dynamic processing technology	Hardware requirements	Advantages	Disadvantages
ORB-SLAM338	3D map (complete SLAM system)	Multi-sensor fusion based on feature points	CPU	High accuracy, robust tracking, multi-map support, and efficient storage; widely adopted in research and industry	Requires sufficient texture for robust feature extraction; performance may degrade in low-texture or repetitive environments
DS-SLAM67	Semantic map	Semantic segmentation and motion consistency check	CPU and lightweight GPU	Strong semantic understanding; motion consistency check improves robustness in dynamic environments	Semantic segmentation network increases computational overhead; generalization to unseen object categories remains challenging
MaskFusion2	Object-level map	Instance segmentation and object-level dynamic modeling	RGB-D camera and GPU	Enables real-time object-aware mapping; integrates semantic labels without requiring known object models	Requires dedicated GPU; instance segmentation network limits frame rate (≈5 Hz); pre-trained models may not cover all object categories
Kimera111	Hybrid map	Real-time semantic segmentation and 3D mesh reconstruction	GPU recommended (CUDA acceleration) for real-time semantic segmentation and 3D mesh reconstruction; can run on CPU-only with reduced performance	Provides real-time metric-semantic mapping and 3D mesh reconstruction; supports dynamic scene graph generation	Semantic segmentation and mesh reconstruction are computationally demanding; requires GPU for real-time performance

“Complete SLAM systems” include front-end tracking, back-end optimization, and loop closure detection. “Representation methods” focus on scene encoding and novel view synthesis.

Table 9.

Research focus and future directions for each map type.

Map type	Core focus	Key technologies	Current challenges	Future directions
3D maps	Geometric reconstruction, spatial representation	Octomap39, NeRF40, 3D GS24	Real-time performance, geometric accuracy, computational efficiency	Lightweight rendering, integration with semantics, online adaptation
Semantic maps	Scene understanding, object categorization	Sensor fusion (Nvblox51, Voxfield53), deep learning (Mask R-CNN, YOLO)	Semantic consistency, generalization, computational overhead	Lightweight networks, domain adaptation, open-vocabulary understanding
Object-level maps	Individual object identity, dynamic attributes	Instance segmentation (MaskFusion2, OA-SLAM80), point cloud processing	GPU dependence, calibration complexity, cross-scenario generalization	Event cameras, part-level semantics, cross-scenario adaptation
Hybrid maps	Multi-modal integration, unified representation	Hierarchical maps, multimodal fusion (Kimera111, HVL-SLAM123)	Data synchronization, algorithm complexity, unified standards	Real-time multimodal fusion, edge deployment, foundation model integration

Table 10 provides a task-to-map selection guide operationalizing the proposed task-oriented taxonomy. For each manipulation task, it specifies the recommended map type and a representative method. This table focuses on conceptual task-to-map mapping rather than quantitative metrics.

Table 10.

Task-oriented map selection: task-level performance.

Manipulation task	Recommended map type	Task-level metric	Representative method	Main limitation
Object grasping	Object-level map	Grasp success rate	MaskFusion2	Requires dedicated GPU
Obstacle avoidance navigation	3D map	Collision rate	HVL-SLAM123	No semantic information
Human–robot interaction	Semantic map	Social compliance rate	DS-SLAM75	Semantic network overhead
Pick-and-place in cluttered scenes	Hybrid map	Completion rate, Operation time	ESLAM120	Computationally demanding

Task-level metrics are reported from the original studies under their respective experimental conditions. They reflect each method’s performance on the specific task and are not intended for direct comparison across different tasks.

Table 11 reports the quantitative performance of each representative method as presented in the original papers. Each metric is annotated with its source dataset (e.g., TUM RGB-D, Replica) for transparency; entries without an explicit dataset are marked as “Not specified”.

Table 11.

Reported quantitative performance of representative methods.

Method	Metric	Value	Dataset
MaskFusion2	RP-RMSE (translational)	0.039–0.093 m/s	Not specified
MaskFusion2	RP-RMSE (rotational)	0.76–3.35 deg/s	Not specified
HVL-SLAM123	AT RMSE	0.035–0.106 m	TUM RGB-D
DS-SLAM75	RPE (translational)	0.0102–0.0333 m	Not specified
DS-SLAM75	RPE (rotational)	0.2690–0.88266 m	Not specified
DS-SLAM75	FPT	37.57 ms +  > 65 ms	Not specified
ESLAM120	AT RMSE	0.0111–0.0247 m	TUM RGB-D
ESLAM120	FPT	0.18 s	Replica
ESLAM120	FPT	0.55 s	ScanNet

Methods were evaluated on different datasets; direct comparison across methods using different datasets is not recommended. “Not specified” indicates that the original study did not explicitly report the dataset associated with the metric.

Table 12 distinguishes SLAM-level metrics from task-level metrics. SLAM-level metrics (e.g., ATE, RPE, RP-RMSE, FPT) evaluate localization accuracy and computational efficiency, while task-level metrics (e.g., grasp success rate, collision rate, completion rate, operation time) assess manipulation performance.

Table 12.

Definition of metric types.

Metric type	Definition	Examples
SLAM-level metrics	Evaluate localization accuracy and computational efficiency	ATE, RPE, RP-RMSE, FPT
Task-level metrics	Evaluate manipulation task completion	Grasp success rate, collision rate, completion rate, operation time

Summary of trade-offs across map types

Based on the hardware requirements and limitations summarized in Table 8, the task-to-map mapping in Fig. 5, and the task suitability analyses in “Task suitability of 3D maps” to “Task suitability of hybrid maps” sections, the following trade-offs can be observed:

1. 3D maps (e.g., ORB-SLAM3³⁸) are computationally efficient and can run on CPU-only hardware, making them suitable for real-time navigation tasks. However, they lack semantic information required for manipulation.

2. Semantic maps (e.g., DS-SLAM⁶⁷) provide scene understanding for human–robot interaction but incur additional GPU overhead.

3. Object-level maps (e.g., MaskFusion²) supply detailed object pose, size, and shape information necessary for grasping and target-specific manipulation, yet require dedicated GPU resources.

4. Hybrid maps (e.g., Kimera¹¹¹) integrate geometric and semantic information to support complex tasks but are the most computationally demanding.

No single map type is optimal across all dimensions. The choice should be guided by task requirements and available hardware. For resource-constrained platforms, such as UAVs or low-cost mobile robots, real-time operation often outweighs representational richness. Table 8 illustrates that 3D maps can operate efficiently on CPU-only systems, supporting fast navigation. In contrast, richer representations—semantic, object-level, or hybrid maps—enhance manipulation performance (e.g., grasping, pick-and-place) but typically require GPU acceleration. Consequently, 3D maps are recommended when strict real-time performance is critical, while richer maps should be adopted only when task demands justify the additional computational resources.

Future prospects

With the rapid advancement of artificial intelligence, visual segmentation and vision-language models (VLMs) have opened new opportunities for robotic mapping in dynamic indoor environments. Visual segmentation models can accurately distinguish between dynamic and static objects, while VLMs enable robots to interpret human commands and environmental descriptions, optimizing maps based on semantic information. However, integrating these models into robot mapping pipelines remains a challenge, particularly due to the complexity of multimodal fusion¹³². Multimodal data fusion algorithms have enormous potential, but several issues must be addressed. We outline five key research directions for future exploration:

Integration of 3D maps with vision-language models

Combining visual and linguistic information allows robots to understand human intentions and execute autonomous tasks. For example, Prompt-RSVQA¹³³ balances visual and linguistic modalities by converting visual features into text, providing language-contextualized understanding. In the 3D domain, 3D visual encoders can extract Mixture of Features (MoF) from map data and convert them into coordinate-annotated textual descriptions. Methods such as MMVP and MoF¹³⁴ integrate self-supervised visual features with multimodal large language models (MLLMs) for enhanced localization. Generative vision-language models¹³⁵ offer cross-lingual, lightweight, and data-efficient representations, improving 3D map encoding while reducing computational costs. Bridging 3D generative scene (GS) rendering with physical executability is also critical. Pipelines like Splat-Nav¹³⁶ enable real-time navigation in dynamic environments by combining semantic GSplat scenes with online replanning (2 Hz) and pose estimation (~ 25 Hz), outperforming NeRF-based methods. Future research should focus on hybrid representations supporting both high-fidelity visualization and physically executable planning, including Gaussian-to-mesh pipelines¹³⁷ and 3D GS-mesh hybrid frameworks exemplified by EnliveningGS¹³⁸.

Optimization of semantic maps for computational efficiency

Semantic maps enhance localization accuracy but introduce computational overhead. Future research should balance these trade-offs by developing lightweight architectures, edge-computing optimizations, and model compression techniques. MobileNet-based networks⁷⁶, efficient transformers⁷⁷, and edge-deployed frameworks⁷⁸ allow real-time semantic perception on resource-constrained platforms. Techniques such as quantization, pruning, and knowledge distillation reduce memory footprint and inference time. For instance, vision-based obstacle avoidance strategies¹³⁹ integrate optical flow with ORB-SLAM3³⁸ to enable CPU-only deployment while maintaining reliable performance. Furthermore, methods like Dual Semantic Correlation Alignment (DSCA)¹⁴⁰ demonstrate effective utilization of semantic information to improve localization across domains.

Enhancing object-level maps with event cameras

Event cameras¹⁴¹ emulate biological retina characteristics, providing high dynamic range, low latency, high temporal resolution, and low power consumption. They enable robust perception in complex, dynamic indoor environments. Key challenges include quantifying trade-offs (e.g., latency vs. power, accuracy vs. sensitivity) and developing native event-driven, bio-inspired systems¹⁴². Sundar et al.¹⁴³ proposed a generalized event camera that preserves scene intensity in a bandwidth-efficient manner, addressing dynamic perception bottlenecks in object-level mapping.

Open-vocabulary and zero-shot understanding

Traditional semantic SLAM systems are restricted to a closed set of object categories, limiting adaptability to novel objects. Integrating VLMs^144–147 and foundation models¹⁴⁸ enables zero-shot recognition and flexible task specification (e.g., “find the small, red container”), supporting dynamic object queries in real time. Research priorities include lightweight open-vocabulary segmentation models for embedded deployment, zero-shot object pose estimation, and natural language-based retrieval from object-level maps.

Real-time multimodal synchronization for hybrid maps

Hybrid maps rely on synchronized multimodal data (e.g., vision, LiDAR, IMU) for accurate representation. Hard real-time synchronization schemes¹⁴⁹ are essential for stable data collection under high computational load. Additionally, multimodal knowledge representation frameworks¹⁵⁰ facilitate the integration of heterogeneous data, enabling efficient reasoning and task execution in dynamic environments.

Addressing these challenges will drive the development of multimodal fusion technologies, providing robust support for map construction and task execution in dynamic indoor environments.

Conclusion

In dynamic indoor environments, selecting an appropriate map type is a prerequisite for robots to perform tasks stably and efficiently. Motivated by this need, this paper presents a comprehensive study of map types for mobile manipulation in such settings. Focusing on robot mobility, four primary map types are examined—3D maps, semantic maps, object-level maps, and hybrid maps—considering their structure, capabilities, and limitations. This study is operationalized through a task-to-map selection guide, which links each map type to specific manipulation tasks and corresponding evaluation metrics. This framework shifts the evaluation paradigm from purely geometric accuracy to task-level performance, aligning directly with the needs of task-oriented SLAM. The key findings are summarized as follows:

3D maps

3D maps are the most widely used, offering high spatial accuracy, efficient rendering, compact storage, and strong perceptual capabilities. They are particularly suitable for navigation-focused tasks due to their computational efficiency. However, challenges remain in real-time performance, sensor dependency, and computational demands for large-scale environments.

Semantic maps

Semantic maps integrate multiple sensors and probabilistic models to provide robots with enhanced understanding of object categories, scene functions, and environmental attributes. They exhibit strong semantic understanding and noise resilience, enabling improved decision-making in complex scenarios. Nevertheless, these maps rely heavily on large amounts of labeled data, high-quality sensors, and substantial computing resources, which limits their generalizability to novel environments.

Object-level maps

Object-level maps generate detailed, semantically annotated object models in real time, providing essential information such as object pose, size, and shape for manipulation tasks. While they can be optimized for embedded platforms, they depend on GPU acceleration, extensive training data, and precise sensor calibration. Consequently, their real-time performance and adaptability in dynamic scenarios are still limited.

Hybrid maps

Hybrid maps combine the advantages of 3D, semantic, and object-level maps, offering a multi-level, multimodal representation that is highly suitable for complex dynamic environments. They improve accuracy, adaptability, and flexibility in mobile manipulation, enabling both geometric precision and semantic reasoning. However, challenges remain in synchronizing real-time multimodal data and establishing unified standards, limiting widespread deployment in real-world applications.

Overall, research on mapping for task execution in dynamic indoor environments plays a pivotal role in advancing mobile manipulation capabilities. While significant progress has been made, widespread deployment of robots in such environments remains a challenge, and numerous technical and practical issues require further investigation.

Author contributions

Xuerou Shen: Conceptualization, Formal Analysis, Methodology, Visualization, Writing—Original Draft, Writing—Review and Editing; Mingzhang Pan: Methodology, Supervision; Ke Liang (Corresponding Author): Conceptualization, Funding Acquisition Resources.

Funding

The authors acknowledge the support of the National Natural Science Foundation of China (NSFC) under Grant No. 62461002.

Data availability

All data generated or analyzed during this study are publicly available to ensure reproducibility.

Declarations

Competing interests

The authors declare no competing interests.