Multimodal Gaussian Splatting Framework with Accurate Image Relocalization for Robotic Inspection

Loading...
Thumbnail Image

Journal Title

Journal ISSN

Volume Title

Publisher

University of Waterloo

Abstract

This research aims to enhance robotic inspection through the use of multimodal 3D Gaussian Splatting (3DGS) and 3DGS-based image relocalization with pose optimization. Robotic inspection systems rely on accurate localization to support automated monitoring of infrastructure such as buildings, tunnels, and bridges. However, existing approaches often rely on low-resolution and sparse point-cloud maps, which limits the accuracy of image relocalization. In addition, pose updates from relocalization are often not well synchronized with downstream applications, reducing the effectiveness of real-time robotic inspection systems. These limitations hinder the reliability and broader adoption of automated robotic inspection in engineering practice. Despite recent progress in visual localization and scene reconstruction, several technical gaps remain. First, there is a representation gap: conventional point cloud maps are often too sparse to provide the geometric and photometric richness required for accurate image relocalization, while neural implicit representations are often computationally expensive and difficult to deploy in real-time robotic systems. Second, there is a coupling gap: 3D mapping and relocalization are typically designed as separate modules, which limits the use of the scene representation to improve the accuracy of downstream relocalization. Third, there is a system gap: relocalization outputs are not always seamlessly integrated into robotic platforms because image relocalization usually introduces delayed pose measurements. This delay reduces the usability of relocalization for real-time robotic inspection tasks, such as immersive inspection applications that require stable and timely pose updates. These gaps motivate the development of a unified framework that jointly addresses scene representation, image relocalization, and system-level pose integration. To address these challenges, this research proposes a novel multimodal 3DGS framework that integrates 3DGS mapping, image relocalization, and pose graph optimization. The framework first employs a LiDAR-visual multimodal 3DGS technique to construct a high-resolution and geometrically accurate 3D scene representation in metric scale. Based on this 3D scene representation, a single-image relocalization method is developed to estimate camera poses using the 3DGS model. In addition, pose graph optimization is introduced to integrate relocalization results with Mixed Reality (MR) headset tracking poses, enabling synchronized and stable pose updates. This research is structured around three main tasks. The first task is to develop a LiDAR-guided 3DGS framework with dense Simultaneous Localization and Mapping (SLAM) priors and LiDAR-visual bundle adjustment for accurate and metric 3DGS map construction. The second task is to design a coarse-to-fine six-degree-of-freedom camera pose estimation method based on the pre-built 3DGS map for accurate image relocalization. The third task is to formulate a pose graph optimization framework that integrates delayed image relocalization poses with MR Vision-inertial odometry (VIO) poses to ensure real-time pose updates. Overall, this research contributes to robotic inspection by improving the above three practical tasks. These contributions can support robotic inspection workflows where a high-definition map is required and reliable pose estimation is needed for visualization, navigation, and spatially referenced inspection. Although this research experiments on an MR headset in the third task, the proposed framework can also be extended to other robotic platforms and infrastructure inspection scenarios that require accurate map-based localization.

Description

Keywords

Citation

Endorsement

Review

Supplemented By

Referenced By