Dynamic Dual-Camera Visual Servo Tracking for Mobile Manipulators

Abstract

Mobile manipulators combine the extended workspace of a mobile platform with the dexterity of a robotic arm, making them well suited to vision-guided tasks in dynamic environments. However, conventional Image-Based Visual Servoing (IBVS) commonly relies on a single camera and is therefore vulnerable to tracking interruptions when the camera–target line of sight becomes obstructed. Dual-camera systems provide complementary viewpoints, but existing approaches often fuse their measurements, switch between them reactively, or assign fixed sensor roles, limiting their ability to coordinate both viewpoints proactively.

This thesis presents a dynamic dual-camera visual servoing framework using an end-effector-mounted camera and a base-mounted camera. A real-time visibility score assigns the cameras to active and auxiliary roles according to the quality of their target observations. The active camera generates the primary IBVS tracking command, while the auxiliary camera uses target segmentation, depth measurements, and predicted image-plane occluder motion to generate a visibility-maintenance command. When the active observation becomes unreliable and the alternate camera provides a better view, the camera roles are exchanged. A two-level Hierarchical Quadratic Programming (HQP) controller maps both camera commands to the generalized velocity of the mobile manipulator. The first level determines the best feasible motion for active-camera tracking, while the second level optimizes auxiliary-camera repositioning while preserving the active-camera velocity achieved at the first level.

The framework is evaluated in two simulated dynamic-occlusion scenarios and in a physical experiment on a 10-DOF mobile manipulator equipped with two RGB-D cameras. In simulation, the proposed controller produced a frame total-occlusion percentage of 1.84%, compared with 9.85% for a fixed-role multi-camera controller, 21.30% for Control Barrier Function visual servoing, and 54.58% for visibility-aware Model Predictive Control. During physical validation, the end-effector and base-camera observations were individually unavailable for 24.58% and 34.58% of the trial, respectively, however, camera coordination limited simultaneous observation loss to 11.67%. The mean active-camera tracking error was 78.41 pixels and the HQP solver required an average of 4.95 ms per control cycle. These results indicate that dynamic camera-role assignment and prioritized viewpoint control can improve camera coordination during target tracking tasks for mobile manipulators.