Active Perception for Robotic Grasping under Occlusion: Semantic-Aware and VLM-Distilled Approaches

Loading...
Thumbnail Image

Journal Title

Journal ISSN

Volume Title

Publisher

University of Waterloo

Abstract

Robotic grasping in cluttered environments is challenging when target objects are heavily occluded, as a single viewpoint is often insufficient for reliable grasp planning. This thesis addresses this problem through two approaches to next-best-view (NBV) planning for active grasping under occlusion. The first approach is a semantic-aware NBV framework that maintains a unified semantic–geometric volumetric representation, integrating geometric and semantic information gain to guide viewpoint selection toward regions most likely to reveal graspable target surfaces. Semantic detections are incrementally accumulated across views through a confidence-weighted update strategy, which adds robustness to false positives and enables autonomous target localization without any prior knowledge of object location. The second approach is a learned NBV policy distilled from Vision-Language Model (VLM) viewpoint preferences. By ranking candidate views offline based on target visibility, occlusion, and grasp feasibility, the VLM provides training labels for a lightweight view scoring network that efficiently selects NBVs at inference without further VLM queries or ray casting. Experiments in simulation and real-world settings validate both approaches, demonstrating that semantic volumetric reasoning and learned VLM preferences are effective and complementary strategies for active perception in robotic grasping.

Description

Keywords

Citation

Endorsement

Review

Supplemented By

Referenced By