Reinforcement Learning Based Motion Planner and Trajectory Tracker for Unmanned Aerial Systems

Abstract

Unmanned aerial vehicles (UAVs) are increasingly used for autonomous tasks such as inspection, maintenance, and search-and-rescue. However, effective trajectory tracking and obstacle avoidance in dynamic environments remain challenging. Traditional optimization-based methods lack adaptability and computational efficiency, motivating the use of reinforcement learning (RL) for UAV control. This thesis explores RL-based UAV trajectory tracking through three key phases: benchmarking RL algorithms, developing a dual-agent RL framework, and leveraging these insights to design a model-free RL approach for aerial continuum manipulators (ACMs).

First, off-policy RL algorithms including DDPG, TD3, SAC, and SoftQ, are benchmarked to evaluate their generalization from simulation to real-world UAV path planning. Results showed DDPG excelling in reward maximization while TD3 provided superior collision avoidance. These insights guided the development of a dual-agent RL framework for UAV trajectory tracking in cluttered environments. The system used two RL agents: one for velocity prediction and another for real-time collision avoidance, leveraging 3D point cloud data to eliminate memory-intensive obstacle representations. Simulated and real-world experiments demonstrated improved trajectory tracking, obstacle avoidance, and adaptability over single-agent and optimization-based approaches.

Building on this, a model-free RL framework for ACMs is introduced, which integrate UAV mobility with continuum robotic arms for dexterous aerial manipulation. Traditional RL struggles with constraint enforcement, leading to unsafe behaviors. To address this, a health-driven RL architecture is proposed that implicitly incorporates constraints through a secondary health reward, ensuring safe and stable operation. Using 3D point cloud data for navigation and a curriculum learning paradigm for scalability, the framework demonstrated superior performance over state-of-the-art RL and optimization techniques in trajectory tracking and constraint adherence.

This thesis advances RL-based aerial control through benchmarking, dual-agent learning, and safe RL integration for complex aerial systems. The findings lay the groundwork for future research in refining metrics, exploring additional algorithms, and incorporating vision-based RL for enhanced perception and decision-making.