Learning to Concurrently Execute Multiple Robotic Tasks – Analysis and Optimization-Based Synthesis

Abstract

In this thesis, we develop and analyze a general framework for learning and concurrently executing multiple robotic control tasks learned within the paradigm of reinforcement learning – each encoded as a value function and executed together as constraints in a pointwise min-norm controller. Applying reinforcement learning to robotic control tasks is an active area of research as it has the potential to generalize to complex tasks. For robots that are redundant – meaning they have the physical capability of executing multiple tasks at the same time – it can be useful to learn to concurrently execute multiple control objectives. Previous and relatively recent work proposes a model-based approach of concurrently executing multiple tasks learned using reinforcent learning by incorporating the learned value functions as constraints in a pointwise min-norm controller. This work, however, does not enforce nor encourage any constraints on the learned value functions themselves, causing there to be no guarantees on how trained tasks execute together. Building on this approach, this thesis proposes ideas for enabling learned value functions to be concurrently executable when combined using the proposed pointwise min-norm controller. Firstly, we define notions of independence, orthogonality, concurrent executability and concurrent controllability to describe the relationships between value functions and their ability to execute together concurrently, based on the Lie derivatives of each value function. We then propose the idea of an “interference cost” which we prove can be used to train a set of value functions to be independent – meaning that their Lie derivatives are linearly independent, making it possible to pick inputs which concurrently execute each task. This interference cost can be used with any reinforcement learning algorithm, but we also propose a variant of fitted value iteration which can be used to efficiently learn such value functions. Additionally, we propose theory that provides necessary and sufficient conditions for value functions to be concurrently executable, using the proposed pointwise min-norm controller, within subsets of the state space. Finally, we test our proposed methods and theorems on simulations involving teams of mobile robots.